Image Captioning Trials¶
In [ ]:
%pip install -q -r requirements.txt
Note: you may need to restart the kernel to use updated packages.
In [ ]:
import tensorflow as tf
from tensorflow.keras import layers, regularizers
from tensorflow.keras.models import Sequential, Model
from onedrivedownloader import download
import os
import shutil
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import PIL
import datetime
import requests
from tensorboard.plugins.hparams import api as hp
import visualkeras
import zipfile
import json
from tqdm import tqdm
import collections
import nltk
from nltk.translate.bleu_score import corpus_bleu
import time
# Make sure to download NLTK data for BLEU if not already installed
nltk.download('punkt')
2024-10-24 09:57:19.196963: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-10-24 09:57:19.210664: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-10-24 09:57:19.214395: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-10-24 09:57:19.224620: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-24 09:57:19.966669: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT [nltk_data] Downloading package punkt to /home/arslane/nltk_data... [nltk_data] Package punkt is already up-to-date!
True
In [ ]:
os.environ['CUDA_VISIBLE_DEVICES'] = "0"
gpus = tf.config.experimental.list_physical_devices('GPU')
print(gpus)
if gpus:
try:
# Enable memory growth for each GPU
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Memory growth must be set before GPUs have been initialized
print(e)
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1729756643.293251 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.332427 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.334481 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.337547 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.339208 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.340815 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.460215 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.462108 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1729756643.463745 9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 2024-10-24 09:57:23.465325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2179 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2050, pci bus id: 0000:01:00.0, compute capability: 8.6
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')] 1 Physical GPUs, 1 Logical GPUs
Loading the COCO dataset¶
In [ ]:
# URLs for the zip files
annotations_url = "https://viacesifr-my.sharepoint.com/:u:/r/personal/salma_turki_viacesi_fr/Documents/COCO%20Dataset/annotations_trainval2014.zip?csf=1&web=1&e=JJ2vXX&download=1"
train2014_url = "https://viacesifr-my.sharepoint.com/:u:/r/personal/salma_turki_viacesi_fr/Documents/COCO%20Dataset/train2014.zip?csf=1&web=1&e=e4CdT1&download=1"
# Local paths
annotation_folder = os.path.join(".", "annotations")
train2014_folder = os.path.join(".", "train2014")
annotation_zip = os.path.join(".", "annotations.zip")
train2014_zip = os.path.join(".", "train2014.zip")
# Check if the folders already exist
if not os.path.exists(annotation_folder):
print("annotations folder not found. Downloading...")
# Download the annotations.zip file
response = requests.get(annotations_url, stream=True)
with open(annotation_zip, 'wb') as f:
shutil.copyfileobj(response.raw, f)
del response
# Unzip the annotations.zip
print("Extracting annotations.zip...")
with zipfile.ZipFile(annotation_zip, 'r') as zip_ref:
zip_ref.extractall(annotation_folder)
# Delete the annotations.zip file
print("Deleting annotations.zip...")
os.remove(annotation_zip)
if not os.path.exists(train2014_folder):
print("train2014 folder not found. Downloading...")
# Download the train2014.zip file
response = requests.get(train2014_url, stream=True)
with open(train2014_zip, 'wb') as f:
shutil.copyfileobj(response.raw, f)
del response
# Unzip the train2014.zip
print("Extracting train2014.zip...")
with zipfile.ZipFile(train2014_zip, 'r') as zip_ref:
zip_ref.extractall(train2014_folder)
# Delete the train2014.zip file
print("Deleting train2014.zip...")
os.remove(train2014_zip)
print("Download and extraction completed.")
Download and extraction completed.
In [ ]:
def get_feature_extraction_model(model_choice='InceptionV3'):
"""
Returns the feature extraction model, preprocessing function, image size, and feature shapes based on the chosen model.
"""
if model_choice == 'InceptionV3':
image_model = tf.keras.applications.InceptionV3(include_top=False, weights='imagenet')
preprocess_input = tf.keras.applications.inception_v3.preprocess_input
img_size = (299, 299) # Image size required for InceptionV3
attention_features_shape = 64 # 8 * 8
features_shape = 2048 # Depth of the feature map
elif model_choice == 'ResNet50':
image_model = tf.keras.applications.ResNet50(include_top=False, weights='imagenet')
preprocess_input = tf.keras.applications.resnet50.preprocess_input
img_size = (224, 224) # Image size required for ResNet50
attention_features_shape = 49 # 7 * 7
features_shape = 2048 # Depth of the feature map
else:
raise ValueError("model_choice must be either 'InceptionV3' or 'ResNet50'")
new_input = image_model.input
hidden_layer = image_model.layers[-1].output
image_features_extract_model = tf.keras.Model(new_input, hidden_layer)
return image_features_extract_model, preprocess_input, img_size, features_shape, attention_features_shape
In [ ]:
model_choice = 'InceptionV3' # Change this to 'ResNet50' to use ResNet
image_features_extract_model, preprocess_input, img_size, features_shape, attention_features_shape = get_feature_extraction_model(model_choice)
In [ ]:
# model_choice = 'ResNet50' # Change this to 'InceptionV3' to use ResNet
# image_features_extract_model, preprocess_input, img_size, features_shape, attention_features_shape = get_feature_extraction_model(model_choice)
In [ ]:
# Annotation file path
annotation_file = os.path.join(annotation_folder, "captions_train2014.json")
# Lecture du fichier d'annotation
with open(annotation_file, 'r') as f:
annotations = json.load(f)
# Group all annotations with the same identifier.
image_path_to_caption = collections.defaultdict(list)
for val in annotations['annotations']:
# mark the beginning and end of each annotation
caption = f"<start> {val['caption']} <end>"
# An image's identifier is part of its access path.
image_path = os.path.join(train2014_folder, 'COCO_train2014_' + '%012d.jpg' % (val['image_id']))
# Add caption to image_path
image_path_to_caption[image_path].append(caption)
# Take first images only
image_paths = list(image_path_to_caption.keys())
train_image_paths = image_paths[:5000]
# List of all annotations
train_captions = []
# List of all duplicated image file names (in number of annotations per image)
img_name_vector = []
for image_path in train_image_paths:
caption_list = image_path_to_caption[image_path]
# Add caption_list to train_captions
train_captions.extend(caption_list)
# Add duplicate image_path len(caption_list) times
img_name_vector.extend([image_path] * len(caption_list))
print(f"Number of images: {len(train_image_paths)}")
Number of images: 5000
In [ ]:
# Function to load and preprocess image
def load_image(image_path):
"""
Load and preprocess the image according to the model (InceptionV3 or ResNet50)
The load_image function has as input the path of an image and as output a pair
pair containing the processed image and its path.
The load_image function performs the following processing:
1. Loads the file corresponding to the path image_path
2. Decodes the image into RGB.
3. Resize image.
4. Normalize image pixels between -1 and 1.
"""
img = tf.io.read_file(image_path)
img = tf.image.decode_jpeg(img, channels=3)
img = tf.image.resize(img, img_size) # Adjust image size based on model
img = preprocess_input(img) # Preprocess input according to selected model
return img, image_path
# Image preprocessing for the dataset
encode_train = sorted(set(img_name_vector))
image_dataset = tf.data.Dataset.from_tensor_slices(encode_train)
image_dataset = image_dataset.map(load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE).batch(8)
# Batch processing for feature extraction
for img, path in tqdm(image_dataset):
batch_features = image_features_extract_model(img)
batch_features = tf.reshape(batch_features, (batch_features.shape[0], -1, batch_features.shape[3]))
for bf, p in zip(batch_features, path):
path_of_feature = p.numpy().decode("utf-8")
np.save(path_of_feature, bf.numpy())
# Display batch features shape
for img, path in image_dataset:
batch_features = image_features_extract_model(img)
print(f"Batch features shape: {batch_features.shape}")
break
0%| | 0/625 [00:00<?, ?it/s]2024-10-24 09:57:26.536374: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 8907 W0000 00:00:1729756646.591262 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.606570 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.608286 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.615191 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.617683 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.619977 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.622065 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.624676 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.626764 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.665406 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.678616 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.681239 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.684857 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.688107 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.691618 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.695449 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.699599 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.702966 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.707117 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.710340 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.713572 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.718296 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.721848 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.726088 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.748091 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.752708 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.757967 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.762400 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.767308 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.776802 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.896441 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.898945 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.901453 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.904448 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.907631 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.910673 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.913774 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.916736 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.920137 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.923969 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.927797 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.931085 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.934055 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.937309 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.941340 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.944640 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.948244 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.952392 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.956594 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.960846 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.990076 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.994263 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756646.999324 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.003095 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.006804 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.010999 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.015235 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.018717 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.023306 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.027882 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.032475 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.037224 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.042793 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.049158 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.055670 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.059968 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.064308 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.081042 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.105462 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.108332 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.111248 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.114200 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.117123 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.120065 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.123665 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.127972 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.131582 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.135459 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.139388 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.143184 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.147132 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.151014 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.154964 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.157875 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.161461 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.165382 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.169866 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.174670 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.179669 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.184566 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.189118 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.200489 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.205915 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.211935 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.217560 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.223027 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.229334 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.236608 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.240464 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.244487 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.248274 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.256531 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.261488 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.273098 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced 2024-10-24 09:57:27.279905: W external/local_tsl/tsl/framework/bfc_allocator.cc:363] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. W0000 00:00:1729756647.307655 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.342451 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.393213 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.396629 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.400391 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.404291 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.408090 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.411815 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.415585 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.419905 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.424291 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.428638 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.432360 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.436927 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.441330 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.445563 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.450110 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.453784 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.458480 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.463494 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.467997 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.472481 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.476093 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.480912 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.485254 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.490550 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.494932 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.501023 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.507152 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.511328 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.515681 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.520856 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.526015 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.531554 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.537258 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.542958 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.548222 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.553837 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.560532 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.567254 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.592521 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.594228 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.595886 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.597521 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.599137 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.600714 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.602320 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.604029 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.605590 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.607172 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.608733 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.610270 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.611881 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.613463 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.615095 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.616769 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.618367 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.620130 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.621728 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.623375 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.625079 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.626719 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.628386 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.630095 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.631805 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.633469 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.635210 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.637007 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.638739 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.640448 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.642327 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.644260 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.645977 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.647689 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.649376 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.651383 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.653270 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.655255 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.657114 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.659007 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.699986 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.701793 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.703533 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.705253 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.706966 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.708710 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.710431 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.712220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.713927 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.715790 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.717590 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.719317 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.721187 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.722901 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.724600 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.726356 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.728203 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.730031 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.731959 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.733830 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.735799 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.737623 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.739539 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.741406 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.743315 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.745173 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.747154 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.749073 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.750971 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.752892 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.754798 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.756630 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.758597 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.760555 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.762964 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.765156 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.767352 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.769327 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.771594 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.773805 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.789430 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.791087 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.792873 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.794687 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.796434 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.798181 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.799861 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.801580 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.803295 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.805144 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.806952 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.808833 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.810504 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.812247 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.814089 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.815891 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.817699 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.819507 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.821479 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.823397 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.825399 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.827464 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.829440 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.831428 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.833479 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.835283 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.837075 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.838832 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.841053 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.843223 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.845185 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.848005 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.850419 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.936180 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.937837 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.939343 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.940873 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.942459 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.943961 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.945486 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.947026 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.948596 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.950158 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.951723 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.953284 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.954786 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.956311 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.957822 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.959331 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.960912 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.962431 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.963934 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.965436 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.966977 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.968508 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.970051 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.971766 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.973375 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.974878 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.976405 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.977955 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.979538 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.981107 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.982695 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.984262 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.985879 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.987485 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.989039 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.990595 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.992316 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.993993 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.995587 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756647.997153 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.012694 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.015153 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.017746 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.020177 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.022719 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.025256 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.027685 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.030390 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.032802 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.036165 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.038799 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.041632 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.043978 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.046288 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.048841 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.051381 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.053860 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.056362 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.059208 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.061655 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.064147 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.066634 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.069553 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.072284 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.074837 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.077563 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.080264 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.083283 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.086247 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.089542 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.092484 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.095777 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.098842 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.102144 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.105845 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.109563 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.112810 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.116944 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.127657 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.156086 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.157600 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.159033 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.160482 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.161990 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.163509 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.165030 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.166512 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.167971 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.169399 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.170854 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.172283 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.173724 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.175177 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.176653 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.178147 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.179621 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.181082 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.182551 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.184036 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.185506 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.187016 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.188620 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.190087 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.191548 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.193038 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.194556 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.200486 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.202124 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.203778 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.205353 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.206850 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.208339 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.209907 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.211720 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.213585 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.215169 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.216738 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.218400 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.221428 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.245066 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.246530 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.247954 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.249408 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.250888 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.252369 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.253820 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.255260 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.256699 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.258173 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.259635 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.261099 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.262531 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.263962 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.265392 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.266861 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.268322 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.269802 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.271266 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.272758 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.274254 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.275751 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.277202 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.278688 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.280188 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.281907 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.283480 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.285009 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.286515 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.288128 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.289703 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.291189 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.292813 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.294639 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.296218 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.297769 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.299431 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.300996 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.302835 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.305816 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.335300 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.336808 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.338293 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.339824 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.341309 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.342821 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.344367 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.345890 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.347385 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.348896 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.350434 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.351903 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.353452 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.354940 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.356472 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.358076 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.359571 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.361111 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.362699 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.364311 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.365954 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.367463 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.369056 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.370690 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.372216 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.373773 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.375403 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.377068 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.378715 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.380373 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.381994 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.383614 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.385359 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.387019 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.389060 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.398219 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.400220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.401981 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.403761 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.407530 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.433178 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.434688 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.436166 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.437636 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.439181 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.440679 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.442220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.443767 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.445279 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.446755 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.448335 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.449849 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.451329 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.452823 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.454321 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.455824 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.457382 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.458972 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.460472 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.462098 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.463636 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.465298 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.466931 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.468459 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.470062 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.471721 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.473244 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.474797 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.476420 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.478051 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.479682 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.481310 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.482967 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.484662 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.486429 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.488176 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.489954 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.493749 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.495769 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.497936 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.532528 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.534076 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.535598 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.537094 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.538648 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.540213 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.541789 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.543386 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.544984 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.546592 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.548100 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.549647 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.551274 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.552805 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.554445 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.556134 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.557656 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.559256 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.560888 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.562396 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.563989 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.565722 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.567308 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.568870 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.570470 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.572147 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.573821 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.575545 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.577161 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.578889 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.580577 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.582262 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.584021 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.585770 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.587549 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.589592 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.591545 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.593510 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.607450 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.609006 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.610522 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.612022 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.613587 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.615172 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.616772 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.618320 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.619920 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.621461 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.623059 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.624722 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.626357 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.627871 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.629564 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.631071 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.632713 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.634298 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.635888 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.637432 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.638937 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.640533 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.642161 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.643894 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.645460 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.647117 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.648852 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.650515 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.652135 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.653775 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.655508 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.657174 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.658928 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.660702 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.662744 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.664479 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.666534 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.668408 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.670289 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.674101 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.708437 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.709989 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.711504 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.713008 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.714578 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.716160 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.717767 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.719320 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.720917 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.722456 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.724060 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.725775 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.727413 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.728962 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.730686 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.732236 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.733834 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.735414 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.737028 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.738570 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.740128 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.741772 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.743425 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.745151 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.746758 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.748474 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.750198 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.751911 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.753610 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.755298 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.757030 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.758725 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.760552 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.762364 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.764590 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.766339 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.768513 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.770379 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.772273 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.776763 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.807125 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.808696 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.810223 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.811725 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.813285 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.814853 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.816421 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.818021 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.819614 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.821213 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.822754 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.824295 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.825926 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.827459 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.829101 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.830787 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.832340 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.833939 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.835665 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.837209 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.838802 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.840530 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.842122 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.843726 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.845356 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.847090 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.848777 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.850506 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.852214 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.853937 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.855634 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.857341 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.859179 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.860917 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.862678 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.864872 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.866825 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.868794 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.905072 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.906672 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.908241 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.909797 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.911438 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.913028 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.914580 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.916181 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.917796 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.919548 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.921161 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.922829 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.924502 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.926267 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.927989 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.929737 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.931447 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.933063 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.934713 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.936366 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.938090 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.939793 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.941506 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.943089 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.944917 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.946586 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.948380 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.950225 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.952117 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.953922 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.955719 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.957690 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.959622 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.961449 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.963340 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.965757 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.967908 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.969976 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.972360 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756648.977506 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.023280 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.024891 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.026497 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.028136 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.029744 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.031367 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.033047 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.034906 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.036510 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.038157 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.039880 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.041492 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.043238 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.044937 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.046516 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.048198 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.050060 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.051741 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.053417 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.055089 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.056841 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.058615 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.060292 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.061936 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.063608 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.065286 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.067126 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.068926 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.070687 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.072563 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.074380 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.076160 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.078170 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.080055 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.081991 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.084454 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.086496 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.088905 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.090942 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.095963 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.154374 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.155871 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.157313 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.158755 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.160169 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.161615 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.163044 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.164481 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.166567 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.168035 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.169535 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.171005 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.172417 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.173832 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.175290 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.176713 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.178139 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.179599 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.181149 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.182616 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.184177 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.185620 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.187072 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.188556 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.190057 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.191538 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.193020 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.194491 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.195993 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.197514 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.198975 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.200494 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.202011 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.203579 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.205118 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.206717 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.208319 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.209814 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.211463 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.213085 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.225407 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.226913 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.228395 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.229944 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.231450 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.232987 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.234564 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.236083 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.237619 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.239179 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.240687 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.242186 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.243705 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.245255 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.246783 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.248311 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.249813 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.251406 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.253030 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.254535 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.256051 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.257567 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.259106 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.260617 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.262118 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.263631 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.265142 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.266787 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.268436 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.270087 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.271898 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.273731 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.275271 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.276971 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.307228 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.308921 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.310593 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.312253 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.313929 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.315699 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.317511 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.319267 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.321020 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.322762 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.324489 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.326224 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.328026 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.329788 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.331594 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.333391 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.335244 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.337145 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.339044 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.340803 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.342529 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.344255 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.346029 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.347982 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.349730 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.351474 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.353236 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.355005 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.356767 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.358543 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.360814 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.362995 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.364734 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.367124 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.369173 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.371115 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.373178 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.375186 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.377846 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.397772 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.399279 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.400750 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.402164 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.403589 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.405061 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.406545 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.407987 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.409499 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.410975 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.412474 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.413955 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.415422 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.416912 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.418371 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.419899 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.421385 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.422840 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.424336 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.425793 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.427319 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.428911 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.430356 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.431788 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.433239 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.434718 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.436180 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.437613 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.439141 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.440830 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.442339 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.443859 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.445400 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.447097 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.448635 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.450319 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.451927 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.465625 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.467102 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.468556 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.469997 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.471435 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.472898 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.474367 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.475887 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.477361 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.478797 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.480242 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.481741 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.483225 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.484682 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.486167 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.487717 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.489261 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.490711 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.492141 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.493571 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.495092 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.496536 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.497981 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.499425 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.500880 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.502320 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.503796 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.505217 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.506864 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.508409 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.509924 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.511458 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.512962 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.514635 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729756649.516150 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced 100%|██████████| 625/625 [01:56<00:00, 5.08it/s]2024-10-24 09:59:23.203779: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence 100%|██████████| 625/625 [01:56<00:00, 5.35it/s]
Batch features shape: (8, 8, 8, 2048)
Pre-processing¶
Annotation pre-processing¶
In [ ]:
# Find the maximum size
def calc_max_length(tensor):
return max(len(t) for t in tensor)
# Choose the 5000 most frequent words in the vocabulary
top_k = 5000
# The Tokenizer class enables text pre-processing for neural networks
tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=top_k,
oov_token="<unk>",
filters='!"#$%&()*+.,-/:;=?@[\]^_`{|}~ ')
# Builds a vocabulary based on the train_captions list
tokenizer.fit_on_texts(train_captions)
# Create the token used to fill annotations to equalize their length
tokenizer.word_index['<pad>'] = 0
tokenizer.index_word[0] = '<pad>'
# Creation of vectors (list of integer tokens) from annotations (list of words)
train_seqs = tokenizer.texts_to_sequences(train_captions)
# Fill each vector up to the maximum annotation length
cap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')
# Calculates the maximum length used to store attention weights
# It will later be used for display during evaluation.
max_length = calc_max_length(train_seqs)
Formation of a training and test set¶
In [ ]:
img_to_cap_vector = collections.defaultdict(list)
# Creation of a dictionary associating image paths (.npy file) with annotations
# # Images are duplicated because there are several annotations per image
print(len(img_name_vector), len(cap_vector))
for img, cap in zip(img_name_vector, cap_vector):
img_to_cap_vector[img].append(cap)
"""
Creation of training and validation datasets
using a random 80-20 split
"""
# Take the keys (names of processed image files), *these will not be duplicated*.
img_keys = list(img_to_cap_vector.keys())
# Dividing clues into training and testing
slice_index = int(len(img_keys)*0.8)
img_name_train_keys, img_name_val_keys = img_keys[:slice_index], img_keys[slice_index:]
"""
Training and test games are in the form of
lists containing mappings:(pre-processed image ---> annotation token(word) )
"""
# Loop to build the training set
img_name_train = []
cap_train = []
for imgt in img_name_train_keys:
capt_len = len(img_to_cap_vector[imgt])
# Duplication of images by number of annotations per image
img_name_train.extend([imgt] * capt_len)
cap_train.extend(img_to_cap_vector[imgt])
# Loop to build the test set
img_name_val = []
cap_val = []
for imgv in img_name_val_keys:
capv_len = len(img_to_cap_vector[imgv])
# Duplication of images by number of annotations per image
img_name_val.extend([imgv] * capv_len)
cap_val.extend(img_to_cap_vector[imgv])
len(img_name_train), len(cap_train), len(img_name_val), len(cap_val)
25011 25011
(20008, 20008, 5003, 5003)
In [ ]:
BATCH_SIZE = 16 # batch size
BUFFER_SIZE = 1000 # buffer size for data mixing
embedding_dim = 256
units = 512 # Hidden layer size in RNN
vocab_size = top_k + 1
num_steps = len(img_name_train) // BATCH_SIZE
# Function that loads numpy files from pre-processed images
def map_func(img_name, cap):
img_tensor = np.load(img_name.decode('utf-8')+'.npy')
return img_tensor, cap
# Creation of a "Tensor "s dataset (used to represent large datasets)
# The dataset is created from "img_name_train" and "cap_train".
dataset = tf.data.Dataset.from_tensor_slices((img_name_train, cap_train))
# Use map to load numpy files (possibly in parallel)
dataset = dataset.map(lambda item1, item2: tf.numpy_function(
map_func, [item1, item2], [tf.float32, tf.int32]),
num_parallel_calls=tf.data.experimental.AUTOTUNE)
# Mixing data and dividing them into batches
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
In [ ]:
import tensorflow as tf
from abc import ABC, abstractmethod
# Base Encoder Class
class BaseEncoder(tf.keras.Model, ABC):
@abstractmethod
def call(self, x):
pass
# Base Decoder Class
class BaseDecoder(tf.keras.Model, ABC):
@abstractmethod
def call(self, x, features, hidden):
pass
CNN¶
In [ ]:
class CNN_Encoder(BaseEncoder):
def __init__(self, embedding_dim):
super(CNN_Encoder, self).__init__()
self.fc = tf.keras.layers.Dense(embedding_dim)
def call(self, x):
x = self.fc(x)
x = tf.nn.relu(x)
return x
Attention mechanisms¶
Bahdanau¶
In [ ]:
class BahdanauAttention(tf.keras.Model):
def __init__(self, units):
super(BahdanauAttention, self).__init__()
# W1 is the hidden layer for image features
self.W1 = tf.keras.layers.Dense(units)
# W2 is the hidden layer for the previous hidden layer
self.W2 = tf.keras.layers.Dense(units)
# V is the output layer that gives a non-normalized score for each image feature
self.V = tf.keras.layers.Dense(1)
def call(self, features, hidden):
hidden_with_time_axis = tf.expand_dims(hidden, 1)
attention_hidden_layer = (tf.nn.tanh(self.W1(features) + self.W2(hidden_with_time_axis)))
# This gives you a non-normalized score for each image feature.
score = self.V(attention_hidden_layer)
attention_weights = tf.nn.softmax(score, axis=1)
context_vector = attention_weights * features
context_vector = tf.reduce_sum(context_vector, axis=1)
return context_vector, attention_weights
RNN¶
GRU¶
1 layer¶
In [ ]:
class RNN_Decoder_GRU(BaseDecoder):
def __init__(self, embedding_dim, units, vocab_size):
super(RNN_Decoder_GRU, self).__init__()
self.units = units
self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
# self.gru = tf.keras.layers.GRU(self.units,
# return_sequences=True,
# return_state=True,
# recurrent_initializer='glorot_uniform')
self.gru = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
self.fc1 = tf.keras.layers.Dense(self.units)
self.fc2 = tf.keras.layers.Dense(vocab_size)
self.attention = BahdanauAttention(self.units)
def call(self, x, features, hidden):
context_vector, attention_weights = self.attention(features, hidden)
# Ensure x has a time dimension
x = self.embedding(x)
if len(x.shape) == 2: # if shape is [batch_size, embedding_dim]
x = tf.expand_dims(x, 1) # expand dims to [batch_size, 1, embedding_dim]
# Concatenate context vector with x
x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
# Pass the concatenated tensor to the GRU
output, state = self.gru(x)
y = self.fc1(output)
y = tf.reshape(y, (-1, y.shape[2])) # Flatten before final Dense layer
y = self.fc2(y)
return y, state, attention_weights
def reset_state(self, batch_size):
return tf.zeros((batch_size, self.units))
3 layers¶
In [ ]:
class RNN_Decoder_GRU_3L(tf.keras.Model):
def __init__(self, embedding_dim, units, vocab_size):
super(RNN_Decoder_GRU_3L, self).__init__()
self.units = units
self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
# self.gru = tf.keras.layers.GRU(self.units, # Taille de la couche cachée du GRU
# return_sequences=True, # retourne la séquence complète de sortie de chaque pas de temps
# return_state=True, # retourne l'état caché de la dernière étape de temps
# recurrent_initializer='glorot_uniform') # glorot_uniform est une initialisation des poids qui permet de mieux converger lors de l'entrainement en utilisant la fonction d'activation relu ce qui fait que les poids sont initialisés de manière à ce que la variance de la sortie soit égale à la variance de l'entrée
# 3 couches de GRU
self.gru1 = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
self.gru2 = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
self.gru3 = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
#Couche dense qui aura pour entrée la sortie du GRU
self.fc1 = tf.keras.layers.Dense(self.units)
# Dernière couche dense
self.fc2 = tf.keras.layers.Dense(vocab_size)
self.attention = BahdanauAttention(self.units)
def call(self, x, features, hidden):
# L'attention est defini par un modèle a part
context_vector, attention_weights = self.attention(features, hidden)
# Passage du mot courant à la couche embedding
x = self.embedding(x)
# Concaténation
x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1) #DONE tf.expand_dims permet de rajouter une dimension à un tenseur à une position donnée
# Passage du vecteur concaténé à la gru
output, state = self.gru1(x)
output, state = self.gru2(output)
output, state = self.gru3(output)
# Couche dense
y = self.fc1(output)
y = tf.reshape(y, (-1, x.shape[2])) # Aplatir le tenseur pour le passer à la couche dense suivante (fc2)
# Couche dense
y = self.fc2(y)
return y, state, attention_weights
def reset_state(self, batch_size):
return tf.zeros((batch_size, self.units))
LSTM¶
In [ ]:
class RNN_Decoder_LSTM(BaseDecoder):
def __init__(self, embedding_dim, units, vocab_size):
super(RNN_Decoder_LSTM, self).__init__()
self.units = units
self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
self.lstm = tf.keras.layers.LSTM(self.units, return_sequences=True, return_state=True)
self.fc1 = tf.keras.layers.Dense(self.units)
self.fc2 = tf.keras.layers.Dense(vocab_size)
self.attention = BahdanauAttention(self.units)
def call(self, x, features, hidden):
context_vector, attention_weights = self.attention(features, hidden)
x = self.embedding(x)
x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
output, hidden_state, cell_state = self.lstm(x)
print("OUTPUT:", output.shape)
y = self.fc1(output)
y = tf.reshape(y, (-1, x.shape[2]))
y = self.fc2(y)
return y, hidden_state, attention_weights
def reset_state(self, batch_size):
return tf.zeros((batch_size, self.units))
Bidirectional LSTM¶
In [ ]:
class RNN_Decoder_Bidirectional_LSTM(BaseDecoder):
def __init__(self, embedding_dim, units, vocab_size):
super(RNN_Decoder_Bidirectional_LSTM, self).__init__()
self.units = units
self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
self.bilstm = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(self.units, return_sequences=True, return_state=True))
# self.lstm = tf.keras.layers.LSTM(self.units, return_sequences=True, return_state=True)
self.fc0 = tf.keras.layers.Dense(self.units)
self.fc1 = tf.keras.layers.Dense(self.units)
self.fc2 = tf.keras.layers.Dense(vocab_size)
self.attention = BahdanauAttention(self.units)
self.fc_hidden = tf.keras.layers.Dense(self.units)
def call(self, x, features, hidden):
context_vector, attention_weights = self.attention(features, hidden)
x = self.embedding(x)
x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
bi_output, forward_h, forward_c, backward_h, backward_c = self.bilstm(x)
bi_output = self.fc0(bi_output)
reduced_hidden = tf.concat([forward_h, backward_h], axis=1)
reduced_hidden = self.fc_hidden(reduced_hidden)
# output, hidden_state, cell_state = self.lstm(bi_output)
# print("OUTPUT:", output.shape)
y = self.fc1(bi_output)
y = tf.reshape(y, (-1, x.shape[2]))
y = self.fc2(y)
return y, reduced_hidden, attention_weights
def reset_state(self, batch_size):
return tf.zeros((batch_size, self.units))
Training¶
In [ ]:
# Create Model
def create_model(encoder_class, decoder_class, embedding_dim, units, vocab_size):
encoder = encoder_class(embedding_dim)
decoder = decoder_class(embedding_dim, units, vocab_size)
return encoder, decoder
# Tokenization and BLEU evaluation functions
def tokenize_captions(captions, tokenizer):
return [tokenizer.texts_to_sequences([cap])[0] for cap in captions]
def decode_predictions(preds, tokenizer):
pred_captions = []
for pred in preds:
decoded_sentence = []
for idx in pred:
if idx == tokenizer.word_index['<end>']:
break
decoded_sentence.append(tokenizer.index_word[idx])
pred_captions.append(decoded_sentence)
return pred_captions
In [ ]:
# Optimiseur ADAM
optimizer = tf.keras.optimizers.Adam() #DONE
# La fonction de perte
# SparseCategoricalCrossentropy est une fonction de perte qui est utilisée pour les problèmes de classification multi-classes. Elle est utilisée lorsque les étiquettes sont des entiers et non pas des vecteurs one-hot.
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True, reduction='none')
def loss_function(real, pred):
mask = tf.math.logical_not(tf.math.equal(real, 0))
loss_ = loss_object(real, pred)
mask = tf.cast(mask, dtype=loss_.dtype)
loss_ *= mask
return tf.reduce_mean(loss_)
In [ ]:
# Validation step on separate validation dataset
loss_plot = []
@tf.function
def validation_step(img_tensor, target, encoder, decoder):
loss = 0
# Initialisation de l'état caché pour chaque batch
hidden = decoder.reset_state(batch_size=target.shape[0])
# Initialiser l'entrée du décodeur
dec_input = tf.expand_dims([tokenizer.word_index['<start>']] * target.shape[0], 1)
with tf.GradientTape() as tape: # Offre la possibilité de calculer le gradient du loss
features = encoder(img_tensor)
for i in range(1, target.shape[1]):
# Prédiction des i'èmes mot du batch avec le décodeur
predictions, hidden, _ = decoder(dec_input, features, hidden)
loss += loss_function(target[:, i], predictions)
# Le mot correct à l'étap i est donné en entrée à l'étape (i+1)
dec_input = tf.expand_dims(target[:, i], 1)
total_loss = (loss / int(target.shape[1])) # Calcul de la perte moyenne par mot du batch courant
trainable_variables = encoder.trainable_variables + decoder.trainable_variables
gradients = tape.gradient(loss, trainable_variables)
optimizer.apply_gradients(zip(gradients, trainable_variables))
return loss, total_loss
Model 1¶
Bidirectional LSTM with Bahdanau attention and ResNet50
In [ ]:
# Optimizer and Checkpoint Management
checkpoint_path = "./checkpoints/bi-lstm-attention"
In [ ]:
def train_model(encoder, decoder, ckpt_manager, start_epoch=0):
# Modified training loop with BLEU score calculation
EPOCHS = 20
for epoch in range(start_epoch, EPOCHS):
start = time.time()
total_loss = 0
for (batch, (img_tensor, target)) in enumerate(dataset):
batch_loss, t_loss = validation_step(img_tensor, target, encoder, decoder)
total_loss += t_loss
if batch % 100 == 0:
print ('Epoch {} Batch {} Loss {:.4f}'.format(
epoch + 1, batch, batch_loss.numpy() / int(target.shape[1])))
# sauvegarde de la perte
loss_plot.append(total_loss / num_steps)
if epoch % 5 == 0:
ckpt_manager.save()
print ('Epoch {} Loss {:.6f}'.format(epoch + 1,
total_loss/num_steps))
print ('Time taken for 1 epoch {} sec\n'.format(time.time() - start))
# Affichage de la courbe d'entrainement
plt.plot(loss_plot)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Loss Plot')
plt.show()
In [ ]:
# Function to evaluate the image captioning model
def evaluate(image, encoder, decoder):
attention_plot = np.zeros((max_length, attention_features_shape))
hidden = decoder.reset_state(batch_size=1)
temp_input = tf.expand_dims(load_image(image)[0], 0)
img_tensor_val = image_features_extract_model(temp_input)
img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0], -1, img_tensor_val.shape[3]))
features = encoder(img_tensor_val)
dec_input = tf.expand_dims([tokenizer.word_index['<start>']], 0)
result = []
for i in range(max_length):
predictions, hidden, attention_weights = decoder(dec_input, features, hidden)
attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()
predicted_id = tf.random.categorical(predictions, 1)[0][0].numpy()
result.append(tokenizer.index_word[predicted_id])
if tokenizer.index_word[predicted_id] == '<end>':
return result, attention_plot
dec_input = tf.expand_dims([predicted_id], 0)
return result, attention_plot
# Function to display the BLEU score
def display_bleu_score(image, result):
references = []
# Display image
image_show = PIL.Image.open(image)
plt.imshow(image_show)
plt.axis('off')
plt.show()
print("\n" + "*" * 60)
print("Predicted Caption :")
print(' '.join(result))
# Find the corresponding references (captions) for this specific image
# Find the corresponding references (captions) for this specific image
for i, img_name in enumerate(img_name_val):
if img_name == image: # Compare with the image being processed
ref = []
for token in cap_val[i]:
if token != 0: # Ignore padding (0)
word = tokenizer.index_word.get(token, '<unk>') # Handle missing tokens
ref.append(word)
references.append(ref)
print("\n" + "*" * 60)
print("References :")
for ref in references[:5]:
print(' '.join(ref))
print("...")
# Convert references for BLEU score evaluation
references_tokenized = [references] # Multiple references for a single image
# Since we're calculating BLEU for a single image, ensure predictions are used as a single list
predictions = [result] # Make the predicted caption a single hypothesis list
# Calculate BLEU scores
bleu_1 = corpus_bleu(references_tokenized, predictions, weights=(1.0, 0, 0, 0))
bleu_2 = corpus_bleu(references_tokenized, predictions, weights=(0.5, 0.5, 0, 0))
bleu_3 = corpus_bleu(references_tokenized, predictions, weights=(0.33, 0.33, 0.33, 0))
bleu_4 = corpus_bleu(references_tokenized, predictions, weights=(0.25, 0.25, 0.25, 0.25))
# Print BLEU scores
print("\n" + "*" * 60)
print(f"BLEU Score :")
print(f"unigram = {bleu_1:.10f}")
print(f"bigram = {bleu_2:.10f}")
print(f"trigram = {bleu_3:.10f}")
print(f"4-gram = {bleu_4:.10f}")
print("*" * 60)
# Function for visualizing attention on the image
def plot_attention(image, result, attention_plot):
temp_image = np.array(PIL.Image.open(image))
fig = plt.figure(figsize=(10, 10))
len_result = len(result)
for l in range(len_result):
temp_att = np.resize(attention_plot[l], (8, 8))
ax = fig.add_subplot(len_result // 2, len_result // 2, l + 1)
ax.set_title(result[l])
img = ax.imshow(temp_image)
ax.imshow(temp_att, cmap='gray', alpha=0.6, extent=img.get_extent())
plt.tight_layout()
plt.show()
In [ ]:
# Instantiate encoder and decoder
encoder, decoder = create_model(CNN_Encoder, RNN_Decoder_Bidirectional_LSTM, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder,
decoder=decoder,
optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
if ckpt_manager.latest_checkpoint:
print(f"Restored from {ckpt_manager.latest_checkpoint}")
print(f"Resuming training from epoch {start_epoch}")
Restored from ./checkpoints/bi-lstm-attention/ckpt-7 Resuming training from epoch 7
In [ ]:
train_model(encoder, decoder, ckpt_manager)
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function. WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.0 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.16 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.17 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.18 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.21 WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function. WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).encoder.fc._kernel WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.0 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.16 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.17 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.18 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.19 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.20 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.21 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.22
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/keras/src/optimizers/base_optimizer.py:731: UserWarning: Gradients do not exist for variables ['kernel', 'kernel', 'kernel', 'bias', 'kernel'] when minimizing the loss. If using `model.compile()`, did you forget to provide a `loss` argument? warnings.warn(
Epoch 1 Batch 0 Loss 1.3308 Epoch 1 Batch 100 Loss 0.6665 Epoch 1 Batch 200 Loss 0.6181 Epoch 1 Batch 300 Loss 0.6507 Epoch 1 Batch 400 Loss 0.6929 Epoch 1 Batch 500 Loss 0.5760 Epoch 1 Batch 600 Loss 0.6114 Epoch 1 Batch 700 Loss 0.6629 Epoch 1 Batch 800 Loss 0.8464 Epoch 1 Batch 900 Loss 0.7834 Epoch 1 Batch 1000 Loss 0.7182 Epoch 1 Batch 1100 Loss 0.5745 Epoch 1 Batch 1200 Loss 0.5548
2024-10-23 14:26:13.988737: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 0.661850 Time taken for 1 epoch 451.702716588974 sec Epoch 2 Batch 0 Loss 0.6918 Epoch 2 Batch 100 Loss 0.6655 Epoch 2 Batch 200 Loss 0.5469 Epoch 2 Batch 300 Loss 0.6484 Epoch 2 Batch 400 Loss 0.7081 Epoch 2 Batch 500 Loss 0.5811 Epoch 2 Batch 600 Loss 0.6283 Epoch 2 Batch 700 Loss 0.5951 Epoch 2 Batch 800 Loss 0.5884 Epoch 2 Batch 900 Loss 0.6524 Epoch 2 Batch 1000 Loss 0.5944 Epoch 2 Batch 1100 Loss 0.6432 Epoch 2 Batch 1200 Loss 0.5875 Epoch 2 Loss 0.588800 Time taken for 1 epoch 404.31619668006897 sec Epoch 3 Batch 0 Loss 0.6631 Epoch 3 Batch 100 Loss 0.5700 Epoch 3 Batch 200 Loss 0.6315 Epoch 3 Batch 300 Loss 0.5168 Epoch 3 Batch 400 Loss 0.6637 Epoch 3 Batch 500 Loss 0.5791 Epoch 3 Batch 600 Loss 0.5642 Epoch 3 Batch 700 Loss 0.5799 Epoch 3 Batch 800 Loss 0.5303 Epoch 3 Batch 900 Loss 0.5454 Epoch 3 Batch 1000 Loss 0.5725 Epoch 3 Batch 1100 Loss 0.5547 Epoch 3 Batch 1200 Loss 0.5921
2024-10-23 14:39:42.814140: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.548848 Time taken for 1 epoch 404.26971983909607 sec Epoch 4 Batch 0 Loss 0.5791 Epoch 4 Batch 100 Loss 0.5735 Epoch 4 Batch 200 Loss 0.5292 Epoch 4 Batch 300 Loss 0.6178 Epoch 4 Batch 400 Loss 0.6346 Epoch 4 Batch 500 Loss 0.4662 Epoch 4 Batch 600 Loss 0.5170 Epoch 4 Batch 700 Loss 0.5487 Epoch 4 Batch 800 Loss 0.5870 Epoch 4 Batch 900 Loss 0.5018 Epoch 4 Batch 1000 Loss 0.4845 Epoch 4 Batch 1100 Loss 0.4999 Epoch 4 Batch 1200 Loss 0.5039 Epoch 4 Loss 0.514951 Time taken for 1 epoch 404.21688580513 sec Epoch 5 Batch 0 Loss 0.5935 Epoch 5 Batch 100 Loss 0.4677 Epoch 5 Batch 200 Loss 0.5188 Epoch 5 Batch 300 Loss 0.4835 Epoch 5 Batch 400 Loss 0.5191 Epoch 5 Batch 500 Loss 0.5512 Epoch 5 Batch 600 Loss 0.5217 Epoch 5 Batch 700 Loss 0.4792 Epoch 5 Batch 800 Loss 0.4835 Epoch 5 Batch 900 Loss 0.5592 Epoch 5 Batch 1000 Loss 0.4922 Epoch 5 Batch 1100 Loss 0.4928 Epoch 5 Batch 1200 Loss 0.5193 Epoch 5 Loss 0.483366 Time taken for 1 epoch 404.2356188297272 sec Epoch 6 Batch 0 Loss 0.5725 Epoch 6 Batch 100 Loss 0.4082 Epoch 6 Batch 200 Loss 0.4265 Epoch 6 Batch 300 Loss 0.4496 Epoch 6 Batch 400 Loss 0.4361 Epoch 6 Batch 500 Loss 0.5292 Epoch 6 Batch 600 Loss 0.3833 Epoch 6 Batch 700 Loss 0.5077 Epoch 6 Batch 800 Loss 0.4557 Epoch 6 Batch 900 Loss 0.5049 Epoch 6 Batch 1000 Loss 0.3994 Epoch 6 Batch 1100 Loss 0.4148 Epoch 6 Batch 1200 Loss 0.4237 Epoch 6 Loss 0.455444 Time taken for 1 epoch 404.40891885757446 sec Epoch 7 Batch 0 Loss 0.4574 Epoch 7 Batch 100 Loss 0.5356 Epoch 7 Batch 200 Loss 0.4975 Epoch 7 Batch 300 Loss 0.3896 Epoch 7 Batch 400 Loss 0.4934 Epoch 7 Batch 500 Loss 0.4106 Epoch 7 Batch 600 Loss 0.4141 Epoch 7 Batch 700 Loss 0.4749 Epoch 7 Batch 800 Loss 0.4093 Epoch 7 Batch 900 Loss 0.5058 Epoch 7 Batch 1000 Loss 0.4484 Epoch 7 Batch 1100 Loss 0.3890 Epoch 7 Batch 1200 Loss 0.3996
2024-10-23 15:06:39.630645: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.431582 Time taken for 1 epoch 403.9550528526306 sec Epoch 8 Batch 0 Loss 0.4493 Epoch 8 Batch 100 Loss 0.4381 Epoch 8 Batch 200 Loss 0.4439 Epoch 8 Batch 300 Loss 0.4573 Epoch 8 Batch 400 Loss 0.4233 Epoch 8 Batch 500 Loss 0.4287 Epoch 8 Batch 600 Loss 0.3951 Epoch 8 Batch 700 Loss 0.4274 Epoch 8 Batch 800 Loss 0.3688 Epoch 8 Batch 900 Loss 0.3192 Epoch 8 Batch 1000 Loss 0.3349 Epoch 8 Batch 1100 Loss 0.4521 Epoch 8 Batch 1200 Loss 0.3516 Epoch 8 Loss 0.409525 Time taken for 1 epoch 404.1138606071472 sec Epoch 9 Batch 0 Loss 0.3865 Epoch 9 Batch 100 Loss 0.3840 Epoch 9 Batch 200 Loss 0.4069 Epoch 9 Batch 300 Loss 0.3551 Epoch 9 Batch 400 Loss 0.4455 Epoch 9 Batch 500 Loss 0.3898 Epoch 9 Batch 600 Loss 0.3437 Epoch 9 Batch 700 Loss 0.3843 Epoch 9 Batch 800 Loss 0.3510 Epoch 9 Batch 900 Loss 0.3901 Epoch 9 Batch 1000 Loss 0.3577 Epoch 9 Batch 1100 Loss 0.3831 Epoch 9 Batch 1200 Loss 0.3395 Epoch 9 Loss 0.389218 Time taken for 1 epoch 403.94647097587585 sec Epoch 10 Batch 0 Loss 0.4170 Epoch 10 Batch 100 Loss 0.3671 Epoch 10 Batch 200 Loss 0.3773 Epoch 10 Batch 300 Loss 0.3489 Epoch 10 Batch 400 Loss 0.3824 Epoch 10 Batch 500 Loss 0.2920 Epoch 10 Batch 600 Loss 0.3427 Epoch 10 Batch 700 Loss 0.4106 Epoch 10 Batch 800 Loss 0.3972 Epoch 10 Batch 900 Loss 0.3768 Epoch 10 Batch 1000 Loss 0.3519 Epoch 10 Batch 1100 Loss 0.3765 Epoch 10 Batch 1200 Loss 0.4347 Epoch 10 Loss 0.371290 Time taken for 1 epoch 404.24859046936035 sec Epoch 11 Batch 0 Loss 0.3070 Epoch 11 Batch 100 Loss 0.3257 Epoch 11 Batch 200 Loss 0.3190 Epoch 11 Batch 300 Loss 0.3252 Epoch 11 Batch 400 Loss 0.3601 Epoch 11 Batch 500 Loss 0.3623 Epoch 11 Batch 600 Loss 0.3718 Epoch 11 Batch 700 Loss 0.3099 Epoch 11 Batch 800 Loss 0.3237 Epoch 11 Batch 900 Loss 0.4156 Epoch 11 Batch 1000 Loss 0.3426 Epoch 11 Batch 1100 Loss 0.3340 Epoch 11 Batch 1200 Loss 0.3582 Epoch 11 Loss 0.357097 Time taken for 1 epoch 404.46499466896057 sec Epoch 12 Batch 0 Loss 0.3334 Epoch 12 Batch 100 Loss 0.4213 Epoch 12 Batch 200 Loss 0.3404 Epoch 12 Batch 300 Loss 0.3463 Epoch 12 Batch 400 Loss 0.2856 Epoch 12 Batch 500 Loss 0.3015 Epoch 12 Batch 600 Loss 0.3505 Epoch 12 Batch 700 Loss 0.3681 Epoch 12 Batch 800 Loss 0.3821 Epoch 12 Batch 900 Loss 0.3475 Epoch 12 Batch 1000 Loss 0.3103 Epoch 12 Batch 1100 Loss 0.3185 Epoch 12 Batch 1200 Loss 0.3402 Epoch 12 Loss 0.344337 Time taken for 1 epoch 403.9861605167389 sec Epoch 13 Batch 0 Loss 0.3339 Epoch 13 Batch 100 Loss 0.3700 Epoch 13 Batch 200 Loss 0.3345 Epoch 13 Batch 300 Loss 0.3919 Epoch 13 Batch 400 Loss 0.3828 Epoch 13 Batch 500 Loss 0.3937 Epoch 13 Batch 600 Loss 0.3208 Epoch 13 Batch 700 Loss 0.3639 Epoch 13 Batch 800 Loss 0.3492 Epoch 13 Batch 900 Loss 0.3345 Epoch 13 Batch 1000 Loss 0.2932 Epoch 13 Batch 1100 Loss 0.3440 Epoch 13 Batch 1200 Loss 0.2918 Epoch 13 Loss 0.329917 Time taken for 1 epoch 403.9970133304596 sec Epoch 14 Batch 0 Loss 0.2767 Epoch 14 Batch 100 Loss 0.3617 Epoch 14 Batch 200 Loss 0.4307 Epoch 14 Batch 300 Loss 0.3563 Epoch 14 Batch 400 Loss 0.3126 Epoch 14 Batch 500 Loss 0.3392 Epoch 14 Batch 600 Loss 0.2766 Epoch 14 Batch 700 Loss 0.3071 Epoch 14 Batch 800 Loss 0.3272 Epoch 14 Batch 900 Loss 0.3577 Epoch 14 Batch 1000 Loss 0.2934 Epoch 14 Batch 1100 Loss 0.3177 Epoch 14 Batch 1200 Loss 0.2823 Epoch 14 Loss 0.320054 Time taken for 1 epoch 403.80691361427307 sec Epoch 15 Batch 0 Loss 0.2944 Epoch 15 Batch 100 Loss 0.3272 Epoch 15 Batch 200 Loss 0.2953 Epoch 15 Batch 300 Loss 0.2562 Epoch 15 Batch 400 Loss 0.3174 Epoch 15 Batch 500 Loss 0.2696 Epoch 15 Batch 600 Loss 0.2852 Epoch 15 Batch 700 Loss 0.3181 Epoch 15 Batch 800 Loss 0.3468 Epoch 15 Batch 900 Loss 0.3092 Epoch 15 Batch 1000 Loss 0.3094 Epoch 15 Batch 1100 Loss 0.3228 Epoch 15 Batch 1200 Loss 0.3343
2024-10-23 16:00:32.177810: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.308469 Time taken for 1 epoch 403.98316717147827 sec Epoch 16 Batch 0 Loss 0.2629 Epoch 16 Batch 100 Loss 0.3160 Epoch 16 Batch 200 Loss 0.2330 Epoch 16 Batch 300 Loss 0.2633 Epoch 16 Batch 400 Loss 0.2905 Epoch 16 Batch 500 Loss 0.2991 Epoch 16 Batch 600 Loss 0.3089 Epoch 16 Batch 700 Loss 0.2852 Epoch 16 Batch 800 Loss 0.3773 Epoch 16 Batch 900 Loss 0.2933 Epoch 16 Batch 1000 Loss 0.2383 Epoch 16 Batch 1100 Loss 0.3361 Epoch 16 Batch 1200 Loss 0.2893 Epoch 16 Loss 0.299219 Time taken for 1 epoch 404.25311255455017 sec Epoch 17 Batch 0 Loss 0.2630 Epoch 17 Batch 100 Loss 0.3076 Epoch 17 Batch 200 Loss 0.2883 Epoch 17 Batch 300 Loss 0.2994 Epoch 17 Batch 400 Loss 0.2915 Epoch 17 Batch 500 Loss 0.2792 Epoch 17 Batch 600 Loss 0.2767 Epoch 17 Batch 700 Loss 0.3116 Epoch 17 Batch 800 Loss 0.2486 Epoch 17 Batch 900 Loss 0.3119 Epoch 17 Batch 1000 Loss 0.3007 Epoch 17 Batch 1100 Loss 0.3087 Epoch 17 Batch 1200 Loss 0.2493 Epoch 17 Loss 0.290037 Time taken for 1 epoch 404.0419964790344 sec Epoch 18 Batch 0 Loss 0.2740 Epoch 18 Batch 100 Loss 0.3151 Epoch 18 Batch 200 Loss 0.2336 Epoch 18 Batch 300 Loss 0.3689 Epoch 18 Batch 400 Loss 0.2776 Epoch 18 Batch 500 Loss 0.2854 Epoch 18 Batch 600 Loss 0.3365 Epoch 18 Batch 700 Loss 0.2895 Epoch 18 Batch 800 Loss 0.3202 Epoch 18 Batch 900 Loss 0.2926 Epoch 18 Batch 1000 Loss 0.2297 Epoch 18 Batch 1100 Loss 0.2938 Epoch 18 Batch 1200 Loss 0.2719 Epoch 18 Loss 0.283209 Time taken for 1 epoch 403.9328944683075 sec Epoch 19 Batch 0 Loss 0.2679 Epoch 19 Batch 100 Loss 0.2579 Epoch 19 Batch 200 Loss 0.2724 Epoch 19 Batch 300 Loss 0.2480 Epoch 19 Batch 400 Loss 0.2536 Epoch 19 Batch 500 Loss 0.2941 Epoch 19 Batch 600 Loss 0.2518 Epoch 19 Batch 700 Loss 0.3054 Epoch 19 Batch 800 Loss 0.2742 Epoch 19 Batch 900 Loss 0.2620 Epoch 19 Batch 1000 Loss 0.2580 Epoch 19 Batch 1100 Loss 0.2310 Epoch 19 Batch 1200 Loss 0.2654 Epoch 19 Loss 0.275977 Time taken for 1 epoch 403.85397386550903 sec Epoch 20 Batch 0 Loss 0.2785 Epoch 20 Batch 100 Loss 0.2603 Epoch 20 Batch 200 Loss 0.2809 Epoch 20 Batch 300 Loss 0.2486 Epoch 20 Batch 400 Loss 0.2204 Epoch 20 Batch 500 Loss 0.2866 Epoch 20 Batch 600 Loss 0.3094 Epoch 20 Batch 700 Loss 0.2318 Epoch 20 Batch 800 Loss 0.2738 Epoch 20 Batch 900 Loss 0.2883 Epoch 20 Batch 1000 Loss 0.2491 Epoch 20 Batch 1100 Loss 0.2722 Epoch 20 Batch 1200 Loss 0.2617 Epoch 20 Loss 0.269723 Time taken for 1 epoch 403.97234559059143 sec
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder, decoder)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
./train2014/COCO_train2014_000000314787.jpg Real Caption: <start> a bathroom with brick walls a toilet and sink <end> Prediction Caption: a very stylish bathroom near an <unk> pan during everything even <unk> pan position <end>
In [ ]:
display_bleu_score(image, result)
************************************************************ Predicted Caption : a newly painted sterile bathroom with the door <end> ************************************************************ References : <start> a white bath tub sitting next to a toilet <end> <start> a tub and toilet in a small bathroom <end> <start> a bathroom with some beige walls and a brown cabinet <end> <start> a small bathroom that has a sink and a toilet <end> <start> entrance to bathroom with shower bathtub sink with vanity and white toilet commode <end> ... ************************************************************ BLEU Score : unigram = 0.3977063630 bigram = 0.2109156497 trigram = 0.0000000000 4-gram = 0.0000000000 ************************************************************
In [ ]:
# play frequency sound when training is done (linux sound)
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Save the Encoder model
encoder.save('models/captioning_models/encoder_resnet_bilstm_model.keras')
# Save the Decoder model
decoder.save('models/captioning_models/decoder1_resnet_bilstm_model.keras')
In [ ]:
def evaluate_average_bleu(encoder, decoder, img_name_val, cap_val, tokenizer):
total_bleu_1 = 0
total_bleu_2 = 0
total_bleu_3 = 0
total_bleu_4 = 0
num_samples = len(img_name_val)
for i in range(num_samples):
image = img_name_val[i]
# Get real caption
real_caption = [[tokenizer.index_word.get(idx, '<unk>') for idx in cap_val[i] if idx not in [0]]]
# Get the predicted caption
result, _ = evaluate(image, encoder, decoder)
# Convert result into format for BLEU score
result_caption = [result]
# Calculate BLEU scores
bleu_1 = corpus_bleu(real_caption, result_caption, weights=(1.0, 0, 0, 0))
bleu_2 = corpus_bleu(real_caption, result_caption, weights=(0.5, 0.5, 0, 0))
bleu_3 = corpus_bleu(real_caption, result_caption, weights=(0.33, 0.33, 0.33, 0))
bleu_4 = corpus_bleu(real_caption, result_caption, weights=(0.25, 0.25, 0.25, 0.25))
# Accumulate BLEU scores
total_bleu_1 += bleu_1
total_bleu_2 += bleu_2
total_bleu_3 += bleu_3
total_bleu_4 += bleu_4
# Optional: print progress every 500 images
if i % 500 == 0:
print(f"Processed {i}/{num_samples} images")
# Calculate the average BLEU score for each n-gram
avg_bleu_1 = total_bleu_1 / num_samples
avg_bleu_2 = total_bleu_2 / num_samples
avg_bleu_3 = total_bleu_3 / num_samples
avg_bleu_4 = total_bleu_4 / num_samples
# Print the average BLEU scores
print("\n" + "*" * 60)
print(f"Average BLEU Score on Validation Dataset:")
print(f"unigram = {avg_bleu_1:.10f}")
print(f"bigram = {avg_bleu_2:.10f}")
print(f"trigram = {avg_bleu_3:.10f}")
print(f"4-gram = {avg_bleu_4:.10f}")
print("*" * 60)
return avg_bleu_1, avg_bleu_2, avg_bleu_3, avg_bleu_4
In [ ]:
# Evaluate the model on the validation dataset
evaluate_average_bleu(encoder, decoder, img_name_val, cap_val, tokenizer)
#
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.0994896786 bigram = 0.0000000000 trigram = 0.0000000000 4-gram = 0.0000000000 ************************************************************
(0.09948967855497265, 4.364591788851547e-155, 3.806644534300383e-204, 9.343453754705764e-232)
Model 2¶
Single Layer GRU with BLEU and ResNet50
In [ ]:
# Instantiate encoder and decoder
encoder_gru_resnet, decoder_gru_resnet = create_model(CNN_Encoder, RNN_Decoder_GRU, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_resnet,
decoder=decoder_gru_resnet,
optimizer=optimizer)
checkpoint_path = "./checkpoints/gru-attention"
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_resnet, decoder_gru_resnet, ckpt_manager)
Epoch 1 Batch 0 Loss 2.0964 Epoch 1 Batch 100 Loss 1.1680 Epoch 1 Batch 200 Loss 0.9441 Epoch 1 Batch 300 Loss 1.0024 Epoch 1 Batch 400 Loss 1.0602 Epoch 1 Batch 500 Loss 1.0049 Epoch 1 Batch 600 Loss 0.8326 Epoch 1 Batch 700 Loss 0.7792 Epoch 1 Batch 800 Loss 0.8974 Epoch 1 Batch 900 Loss 0.7943 Epoch 1 Batch 1000 Loss 0.9234 Epoch 1 Batch 1100 Loss 0.8752 Epoch 1 Batch 1200 Loss 0.8474
2024-10-23 19:13:16.528266: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 0.943236 Time taken for 1 epoch 234.17758750915527 sec Epoch 2 Batch 0 Loss 1.0063 Epoch 2 Batch 100 Loss 0.9182 Epoch 2 Batch 200 Loss 0.8619 Epoch 2 Batch 300 Loss 0.7751 Epoch 2 Batch 400 Loss 0.8642 Epoch 2 Batch 500 Loss 0.8193 Epoch 2 Batch 600 Loss 0.7891 Epoch 2 Batch 700 Loss 0.7564 Epoch 2 Batch 800 Loss 0.7790 Epoch 2 Batch 900 Loss 0.6983 Epoch 2 Batch 1000 Loss 0.8320 Epoch 2 Batch 1100 Loss 0.7041 Epoch 2 Batch 1200 Loss 0.6296 Epoch 2 Loss 0.790500 Time taken for 1 epoch 176.20997762680054 sec Epoch 3 Batch 0 Loss 0.7597 Epoch 3 Batch 100 Loss 0.7468 Epoch 3 Batch 200 Loss 0.6478 Epoch 3 Batch 300 Loss 0.7488 Epoch 3 Batch 400 Loss 0.7710 Epoch 3 Batch 500 Loss 0.6730 Epoch 3 Batch 600 Loss 0.6807 Epoch 3 Batch 700 Loss 0.6980 Epoch 3 Batch 800 Loss 0.7633 Epoch 3 Batch 900 Loss 0.6081 Epoch 3 Batch 1000 Loss 0.5709 Epoch 3 Batch 1100 Loss 0.7082 Epoch 3 Batch 1200 Loss 0.7064
2024-10-23 19:19:09.036911: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.728305 Time taken for 1 epoch 176.13086009025574 sec Epoch 4 Batch 0 Loss 0.7767 Epoch 4 Batch 100 Loss 0.7977 Epoch 4 Batch 200 Loss 0.7996 Epoch 4 Batch 300 Loss 0.6901 Epoch 4 Batch 400 Loss 0.9279 Epoch 4 Batch 500 Loss 0.6875 Epoch 4 Batch 600 Loss 0.6675 Epoch 4 Batch 700 Loss 0.6429 Epoch 4 Batch 800 Loss 0.5904 Epoch 4 Batch 900 Loss 0.7088 Epoch 4 Batch 1000 Loss 0.6508 Epoch 4 Batch 1100 Loss 0.5496 Epoch 4 Batch 1200 Loss 0.5898 Epoch 4 Loss 0.681548 Time taken for 1 epoch 176.12795042991638 sec Epoch 5 Batch 0 Loss 0.7248 Epoch 5 Batch 100 Loss 0.5946 Epoch 5 Batch 200 Loss 0.6493 Epoch 5 Batch 300 Loss 0.6985 Epoch 5 Batch 400 Loss 0.6287 Epoch 5 Batch 500 Loss 0.6723 Epoch 5 Batch 600 Loss 0.6206 Epoch 5 Batch 700 Loss 0.6803 Epoch 5 Batch 800 Loss 0.6481 Epoch 5 Batch 900 Loss 0.5847 Epoch 5 Batch 1000 Loss 0.6799 Epoch 5 Batch 1100 Loss 0.7091 Epoch 5 Batch 1200 Loss 0.6551 Epoch 5 Loss 0.642496 Time taken for 1 epoch 175.99731302261353 sec Epoch 6 Batch 0 Loss 0.6666 Epoch 6 Batch 100 Loss 0.6303 Epoch 6 Batch 200 Loss 0.6445 Epoch 6 Batch 300 Loss 0.6242 Epoch 6 Batch 400 Loss 0.4923 Epoch 6 Batch 500 Loss 0.4911 Epoch 6 Batch 600 Loss 0.6148 Epoch 6 Batch 700 Loss 0.6318 Epoch 6 Batch 800 Loss 0.5802 Epoch 6 Batch 900 Loss 0.6092 Epoch 6 Batch 1000 Loss 0.6573 Epoch 6 Batch 1100 Loss 0.6447 Epoch 6 Batch 1200 Loss 0.5740 Epoch 6 Loss 0.606568 Time taken for 1 epoch 176.03090715408325 sec Epoch 7 Batch 0 Loss 0.6503 Epoch 7 Batch 100 Loss 0.5569 Epoch 7 Batch 200 Loss 0.6204 Epoch 7 Batch 300 Loss 0.5214 Epoch 7 Batch 400 Loss 0.6260 Epoch 7 Batch 500 Loss 0.6618 Epoch 7 Batch 600 Loss 0.6330 Epoch 7 Batch 700 Loss 0.5900 Epoch 7 Batch 800 Loss 0.5842 Epoch 7 Batch 900 Loss 0.5461 Epoch 7 Batch 1000 Loss 0.5304 Epoch 7 Batch 1100 Loss 0.5495 Epoch 7 Batch 1200 Loss 0.5432
2024-10-23 19:30:53.239752: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.573152 Time taken for 1 epoch 176.04660749435425 sec Epoch 8 Batch 0 Loss 0.6648 Epoch 8 Batch 100 Loss 0.5965 Epoch 8 Batch 200 Loss 0.5658 Epoch 8 Batch 300 Loss 0.5529 Epoch 8 Batch 400 Loss 0.6312 Epoch 8 Batch 500 Loss 0.5119 Epoch 8 Batch 600 Loss 0.4763 Epoch 8 Batch 700 Loss 0.5234 Epoch 8 Batch 800 Loss 0.5312 Epoch 8 Batch 900 Loss 0.5735 Epoch 8 Batch 1000 Loss 0.4996 Epoch 8 Batch 1100 Loss 0.5005 Epoch 8 Batch 1200 Loss 0.5560 Epoch 8 Loss 0.541674 Time taken for 1 epoch 176.0696837902069 sec Epoch 9 Batch 0 Loss 0.5775 Epoch 9 Batch 100 Loss 0.4871 Epoch 9 Batch 200 Loss 0.5153 Epoch 9 Batch 300 Loss 0.4783 Epoch 9 Batch 400 Loss 0.4683 Epoch 9 Batch 500 Loss 0.5768 Epoch 9 Batch 600 Loss 0.4385 Epoch 9 Batch 700 Loss 0.4511 Epoch 9 Batch 800 Loss 0.5406 Epoch 9 Batch 900 Loss 0.5116 Epoch 9 Batch 1000 Loss 0.5508 Epoch 9 Batch 1100 Loss 0.4673 Epoch 9 Batch 1200 Loss 0.4499 Epoch 9 Loss 0.512716 Time taken for 1 epoch 175.94577312469482 sec Epoch 10 Batch 0 Loss 0.4741 Epoch 10 Batch 100 Loss 0.4460 Epoch 10 Batch 200 Loss 0.5673 Epoch 10 Batch 300 Loss 0.5252 Epoch 10 Batch 400 Loss 0.4914 Epoch 10 Batch 500 Loss 0.4032 Epoch 10 Batch 600 Loss 0.5410 Epoch 10 Batch 700 Loss 0.5118 Epoch 10 Batch 800 Loss 0.4926 Epoch 10 Batch 900 Loss 0.4248 Epoch 10 Batch 1000 Loss 0.4502 Epoch 10 Batch 1100 Loss 0.5084 Epoch 10 Batch 1200 Loss 0.3949 Epoch 10 Loss 0.484986 Time taken for 1 epoch 175.99557375907898 sec Epoch 11 Batch 0 Loss 0.3847 Epoch 11 Batch 100 Loss 0.4137 Epoch 11 Batch 200 Loss 0.3729 Epoch 11 Batch 300 Loss 0.4496 Epoch 11 Batch 400 Loss 0.4737 Epoch 11 Batch 500 Loss 0.4954 Epoch 11 Batch 600 Loss 0.5368 Epoch 11 Batch 700 Loss 0.5587 Epoch 11 Batch 800 Loss 0.5428 Epoch 11 Batch 900 Loss 0.4295 Epoch 11 Batch 1000 Loss 0.4540 Epoch 11 Batch 1100 Loss 0.4445 Epoch 11 Batch 1200 Loss 0.5133 Epoch 11 Loss 0.460534 Time taken for 1 epoch 176.05313158035278 sec Epoch 12 Batch 0 Loss 0.5090 Epoch 12 Batch 100 Loss 0.3912 Epoch 12 Batch 200 Loss 0.4012 Epoch 12 Batch 300 Loss 0.4587 Epoch 12 Batch 400 Loss 0.4540 Epoch 12 Batch 500 Loss 0.3738 Epoch 12 Batch 600 Loss 0.4764 Epoch 12 Batch 700 Loss 0.5041 Epoch 12 Batch 800 Loss 0.4003 Epoch 12 Batch 900 Loss 0.4102 Epoch 12 Batch 1000 Loss 0.4880 Epoch 12 Batch 1100 Loss 0.4182 Epoch 12 Batch 1200 Loss 0.4300 Epoch 12 Loss 0.437338 Time taken for 1 epoch 176.01339483261108 sec Epoch 13 Batch 0 Loss 0.4201 Epoch 13 Batch 100 Loss 0.3929 Epoch 13 Batch 200 Loss 0.4307 Epoch 13 Batch 300 Loss 0.3881 Epoch 13 Batch 400 Loss 0.4916 Epoch 13 Batch 500 Loss 0.4010 Epoch 13 Batch 600 Loss 0.3226 Epoch 13 Batch 700 Loss 0.3739 Epoch 13 Batch 800 Loss 0.4380 Epoch 13 Batch 900 Loss 0.3780 Epoch 13 Batch 1000 Loss 0.4192 Epoch 13 Batch 1100 Loss 0.4132 Epoch 13 Batch 1200 Loss 0.3852 Epoch 13 Loss 0.416540 Time taken for 1 epoch 175.97470211982727 sec Epoch 14 Batch 0 Loss 0.3983 Epoch 14 Batch 100 Loss 0.3778 Epoch 14 Batch 200 Loss 0.4336 Epoch 14 Batch 300 Loss 0.5072 Epoch 14 Batch 400 Loss 0.4587 Epoch 14 Batch 500 Loss 0.4077 Epoch 14 Batch 600 Loss 0.3997 Epoch 14 Batch 700 Loss 0.4283 Epoch 14 Batch 800 Loss 0.3579 Epoch 14 Batch 900 Loss 0.4148 Epoch 14 Batch 1000 Loss 0.4393 Epoch 14 Batch 1100 Loss 0.3846 Epoch 14 Batch 1200 Loss 0.4251 Epoch 14 Loss 0.397899 Time taken for 1 epoch 175.8241903781891 sec Epoch 15 Batch 0 Loss 0.3710 Epoch 15 Batch 100 Loss 0.3208 Epoch 15 Batch 200 Loss 0.3491 Epoch 15 Batch 300 Loss 0.3646 Epoch 15 Batch 400 Loss 0.3877 Epoch 15 Batch 500 Loss 0.3793 Epoch 15 Batch 600 Loss 0.3434 Epoch 15 Batch 700 Loss 0.4042 Epoch 15 Batch 800 Loss 0.4302 Epoch 15 Batch 900 Loss 0.4447 Epoch 15 Batch 1000 Loss 0.3603 Epoch 15 Batch 1100 Loss 0.3325 Epoch 15 Batch 1200 Loss 0.4446
2024-10-23 19:54:20.998139: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.380666 Time taken for 1 epoch 175.88196086883545 sec Epoch 16 Batch 0 Loss 0.3584 Epoch 16 Batch 100 Loss 0.3380 Epoch 16 Batch 200 Loss 0.3790 Epoch 16 Batch 300 Loss 0.4311 Epoch 16 Batch 400 Loss 0.3487 Epoch 16 Batch 500 Loss 0.3701 Epoch 16 Batch 600 Loss 0.4060 Epoch 16 Batch 700 Loss 0.3413 Epoch 16 Batch 800 Loss 0.4320 Epoch 16 Batch 900 Loss 0.3482 Epoch 16 Batch 1000 Loss 0.3454 Epoch 16 Batch 1100 Loss 0.3557 Epoch 16 Batch 1200 Loss 0.4170 Epoch 16 Loss 0.365070 Time taken for 1 epoch 176.03572297096252 sec Epoch 17 Batch 0 Loss 0.3553 Epoch 17 Batch 100 Loss 0.3576 Epoch 17 Batch 200 Loss 0.3852 Epoch 17 Batch 300 Loss 0.3946 Epoch 17 Batch 400 Loss 0.3786 Epoch 17 Batch 500 Loss 0.3256 Epoch 17 Batch 600 Loss 0.3781 Epoch 17 Batch 700 Loss 0.3641 Epoch 17 Batch 800 Loss 0.3115 Epoch 17 Batch 900 Loss 0.3560 Epoch 17 Batch 1000 Loss 0.3803 Epoch 17 Batch 1100 Loss 0.3643 Epoch 17 Batch 1200 Loss 0.3805 Epoch 17 Loss 0.351258 Time taken for 1 epoch 175.90720772743225 sec Epoch 18 Batch 0 Loss 0.3787 Epoch 18 Batch 100 Loss 0.3134 Epoch 18 Batch 200 Loss 0.3316 Epoch 18 Batch 300 Loss 0.3373 Epoch 18 Batch 400 Loss 0.2838 Epoch 18 Batch 500 Loss 0.3490 Epoch 18 Batch 600 Loss 0.3052 Epoch 18 Batch 700 Loss 0.2989 Epoch 18 Batch 800 Loss 0.3126 Epoch 18 Batch 900 Loss 0.3357 Epoch 18 Batch 1000 Loss 0.3329 Epoch 18 Batch 1100 Loss 0.3369 Epoch 18 Batch 1200 Loss 0.3137 Epoch 18 Loss 0.338591 Time taken for 1 epoch 176.1427402496338 sec Epoch 19 Batch 0 Loss 0.3480 Epoch 19 Batch 100 Loss 0.2945 Epoch 19 Batch 200 Loss 0.3361 Epoch 19 Batch 300 Loss 0.3796 Epoch 19 Batch 400 Loss 0.3780 Epoch 19 Batch 500 Loss 0.2687 Epoch 19 Batch 600 Loss 0.3110 Epoch 19 Batch 700 Loss 0.3021 Epoch 19 Batch 800 Loss 0.3345 Epoch 19 Batch 900 Loss 0.3549 Epoch 19 Batch 1000 Loss 0.3258 Epoch 19 Batch 1100 Loss 0.2691 Epoch 19 Batch 1200 Loss 0.4158 Epoch 19 Loss 0.327115 Time taken for 1 epoch 176.8774073123932 sec Epoch 20 Batch 0 Loss 0.3616 Epoch 20 Batch 100 Loss 0.3117 Epoch 20 Batch 200 Loss 0.3318 Epoch 20 Batch 300 Loss 0.2756 Epoch 20 Batch 400 Loss 0.3554 Epoch 20 Batch 500 Loss 0.2905 Epoch 20 Batch 600 Loss 0.2984 Epoch 20 Batch 700 Loss 0.3065 Epoch 20 Batch 800 Loss 0.3337 Epoch 20 Batch 900 Loss 0.2904 Epoch 20 Batch 1000 Loss 0.3287 Epoch 20 Batch 1100 Loss 0.3136 Epoch 20 Batch 1200 Loss 0.3474 Epoch 20 Loss 0.316204 Time taken for 1 epoch 176.05854892730713 sec
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder_gru_resnet, decoder_gru_resnet)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
display_bleu_score(image, result)
./train2014/COCO_train2014_000000461189.jpg Real Caption: <start> a kitchen with an oven and a table <end> Prediction Caption: a view of a large kitchen with wooden cabinets and a wooden floor and black countertops counter space and white cabinets <end>
************************************************************ Predicted Caption : a view of a large kitchen with wooden cabinets and a wooden floor and black countertops counter space and white cabinets <end> ************************************************************ References : <start> a kitchen with an oven and a table <end> <start> a kitchen with <unk> colored cupboards and a silver oven <end> <start> the interior of a kitchen with multiple lighting and wood floors <end> <start> a dimly lit kitchen with a modern dark decor <end> <start> a large kitchen with black cabinets and silver stove <end> ... ************************************************************ BLEU Score : unigram = 0.4545454545 bigram = 0.3603749851 trigram = 0.2384833484 4-gram = 0.1616921435 ************************************************************
In [ ]:
encoder_gru_resnet.save('models/captioning_models/encoder_resnet_gru_model.keras')
decoder_gru_resnet.save('models/captioning_models/decoder_resnet_gru_model.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_resnet, decoder_gru_resnet, img_name_val, cap_val, tokenizer)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 2-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg) /home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 3-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg) /home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 4-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg)
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.1034515997 bigram = 0.0000000000 trigram = 0.0000000000 4-gram = 0.0000000000 ************************************************************
(0.10345159969758377, 4.439541344333561e-155, 3.857526580399094e-204, 9.458452165801623e-232)
Model 3¶
3 Layers GRU with BLEU and ResNet50
In [ ]:
# Instantiate encoder and decoder
encoder_gru_L3_resnet, decoder_gru_L3_resnet = create_model(CNN_Encoder, RNN_Decoder_GRU_3L, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_L3_resnet,
decoder=decoder_gru_L3_resnet,
optimizer=optimizer)
checkpoint_path = "./checkpoints/gru-l3-attention"
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_L3_resnet, decoder_gru_L3_resnet, ckpt_manager)
Epoch 1 Batch 0 Loss 1.8800 Epoch 1 Batch 100 Loss 1.3634 Epoch 1 Batch 200 Loss 1.2763 Epoch 1 Batch 300 Loss 1.2612 Epoch 1 Batch 400 Loss 1.3582 Epoch 1 Batch 500 Loss 1.4150 Epoch 1 Batch 600 Loss 1.1775 Epoch 1 Batch 700 Loss 1.3302 Epoch 1 Batch 800 Loss 1.0843 Epoch 1 Batch 900 Loss 1.1823 Epoch 1 Batch 1000 Loss 1.2612 Epoch 1 Batch 1100 Loss 1.4069 Epoch 1 Batch 1200 Loss 1.3640
2024-10-23 21:08:49.586463: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 1.275946 Time taken for 1 epoch 392.353985786438 sec Epoch 2 Batch 0 Loss 1.4340 Epoch 2 Batch 100 Loss 1.4314 Epoch 2 Batch 200 Loss 1.0692 Epoch 2 Batch 300 Loss 1.0366 Epoch 2 Batch 400 Loss 1.0671 Epoch 2 Batch 500 Loss 0.9356 Epoch 2 Batch 600 Loss 0.9546 Epoch 2 Batch 700 Loss 0.9686 Epoch 2 Batch 800 Loss 0.9334 Epoch 2 Batch 900 Loss 0.8734 Epoch 2 Batch 1000 Loss 0.9508 Epoch 2 Batch 1100 Loss 0.9545 Epoch 2 Batch 1200 Loss 0.9678 Epoch 2 Loss 1.018137 Time taken for 1 epoch 261.60807394981384 sec Epoch 3 Batch 0 Loss 0.8519 Epoch 3 Batch 100 Loss 0.9725 Epoch 3 Batch 200 Loss 0.7563 Epoch 3 Batch 300 Loss 0.9703 Epoch 3 Batch 400 Loss 0.9373 Epoch 3 Batch 500 Loss 0.8537 Epoch 3 Batch 600 Loss 0.8569 Epoch 3 Batch 700 Loss 0.9496 Epoch 3 Batch 800 Loss 1.0092 Epoch 3 Batch 900 Loss 0.7985 Epoch 3 Batch 1000 Loss 0.8884 Epoch 3 Batch 1100 Loss 0.7937 Epoch 3 Batch 1200 Loss 0.9104
2024-10-23 21:17:34.670808: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.863571 Time taken for 1 epoch 263.3019468784332 sec Epoch 4 Batch 0 Loss 0.7813 Epoch 4 Batch 100 Loss 0.8211 Epoch 4 Batch 200 Loss 0.7291 Epoch 4 Batch 300 Loss 0.6982 Epoch 4 Batch 400 Loss 0.9262 Epoch 4 Batch 500 Loss 0.7782 Epoch 4 Batch 600 Loss 0.7966 Epoch 4 Batch 700 Loss 0.9356 Epoch 4 Batch 800 Loss 0.7733 Epoch 4 Batch 900 Loss 0.7418 Epoch 4 Batch 1000 Loss 0.8871 Epoch 4 Batch 1100 Loss 0.7817 Epoch 4 Batch 1200 Loss 0.6912 Epoch 4 Loss 0.806589 Time taken for 1 epoch 264.0402846336365 sec Epoch 5 Batch 0 Loss 0.8809 Epoch 5 Batch 100 Loss 0.7652 Epoch 5 Batch 200 Loss 0.8330 Epoch 5 Batch 300 Loss 0.8552 Epoch 5 Batch 400 Loss 0.8630 Epoch 5 Batch 500 Loss 0.9165 Epoch 5 Batch 600 Loss 0.6650 Epoch 5 Batch 700 Loss 0.9382 Epoch 5 Batch 800 Loss 0.7748 Epoch 5 Batch 900 Loss 0.7450 Epoch 5 Batch 1000 Loss 0.7221 Epoch 5 Batch 1100 Loss 0.6803 Epoch 5 Batch 1200 Loss 0.8076 Epoch 5 Loss 0.769270 Time taken for 1 epoch 263.58511996269226 sec Epoch 6 Batch 0 Loss 0.9396 Epoch 6 Batch 100 Loss 0.7709 Epoch 6 Batch 200 Loss 0.7401 Epoch 6 Batch 300 Loss 0.8272 Epoch 6 Batch 400 Loss 0.6988 Epoch 6 Batch 500 Loss 0.6924 Epoch 6 Batch 600 Loss 0.6814 Epoch 6 Batch 700 Loss 0.8023 Epoch 6 Batch 800 Loss 0.6797 Epoch 6 Batch 900 Loss 0.6853 Epoch 6 Batch 1000 Loss 0.6215 Epoch 6 Batch 1100 Loss 0.7544 Epoch 6 Batch 1200 Loss 0.7260 Epoch 6 Loss 0.738621 Time taken for 1 epoch 262.6053535938263 sec Epoch 7 Batch 0 Loss 0.7672 Epoch 7 Batch 100 Loss 0.7677 Epoch 7 Batch 200 Loss 0.6071 Epoch 7 Batch 300 Loss 0.7035 Epoch 7 Batch 400 Loss 0.6916 Epoch 7 Batch 500 Loss 0.6411 Epoch 7 Batch 600 Loss 0.7217 Epoch 7 Batch 700 Loss 0.7668 Epoch 7 Batch 800 Loss 0.7479 Epoch 7 Batch 900 Loss 0.7643 Epoch 7 Batch 1000 Loss 0.8137 Epoch 7 Batch 1100 Loss 0.7446 Epoch 7 Batch 1200 Loss 0.6719
2024-10-23 21:35:07.461676: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.712677 Time taken for 1 epoch 262.56003999710083 sec Epoch 8 Batch 0 Loss 0.8302 Epoch 8 Batch 100 Loss 0.7190 Epoch 8 Batch 200 Loss 0.6850 Epoch 8 Batch 300 Loss 0.6612 Epoch 8 Batch 400 Loss 0.6782 Epoch 8 Batch 500 Loss 0.7693 Epoch 8 Batch 600 Loss 0.6354 Epoch 8 Batch 700 Loss 0.7167 Epoch 8 Batch 800 Loss 0.6314 Epoch 8 Batch 900 Loss 0.7537 Epoch 8 Batch 1000 Loss 0.6188 Epoch 8 Batch 1100 Loss 0.6421 Epoch 8 Batch 1200 Loss 0.6549 Epoch 8 Loss 0.688417 Time taken for 1 epoch 263.40365076065063 sec Epoch 9 Batch 0 Loss 0.6985 Epoch 9 Batch 100 Loss 0.7271 Epoch 9 Batch 200 Loss 0.5734 Epoch 9 Batch 300 Loss 0.5354 Epoch 9 Batch 400 Loss 0.6367 Epoch 9 Batch 500 Loss 0.6057 Epoch 9 Batch 600 Loss 0.5762 Epoch 9 Batch 700 Loss 0.6228 Epoch 9 Batch 800 Loss 0.6960 Epoch 9 Batch 900 Loss 0.7059 Epoch 9 Batch 1000 Loss 0.7739 Epoch 9 Batch 1100 Loss 0.6428 Epoch 9 Batch 1200 Loss 0.7124 Epoch 9 Loss 0.666510 Time taken for 1 epoch 261.74695348739624 sec Epoch 10 Batch 0 Loss 0.7298 Epoch 10 Batch 100 Loss 0.7012 Epoch 10 Batch 200 Loss 0.5880 Epoch 10 Batch 300 Loss 0.5870 Epoch 10 Batch 400 Loss 0.7663 Epoch 10 Batch 500 Loss 0.5953 Epoch 10 Batch 600 Loss 0.6331 Epoch 10 Batch 700 Loss 0.6985 Epoch 10 Batch 800 Loss 0.5709 Epoch 10 Batch 900 Loss 0.6214 Epoch 10 Batch 1000 Loss 0.5542 Epoch 10 Batch 1100 Loss 0.6052 Epoch 10 Batch 1200 Loss 0.6134 Epoch 10 Loss 0.646228 Time taken for 1 epoch 261.6729748249054 sec Epoch 11 Batch 0 Loss 0.7387 Epoch 11 Batch 100 Loss 0.6400 Epoch 11 Batch 200 Loss 0.6309 Epoch 11 Batch 300 Loss 0.6331 Epoch 11 Batch 400 Loss 0.6732 Epoch 11 Batch 500 Loss 0.6432 Epoch 11 Batch 600 Loss 0.5667 Epoch 11 Batch 700 Loss 0.6931 Epoch 11 Batch 800 Loss 0.4984 Epoch 11 Batch 900 Loss 0.6606 Epoch 11 Batch 1000 Loss 0.6421 Epoch 11 Batch 1100 Loss 0.7174 Epoch 11 Batch 1200 Loss 0.6267 Epoch 11 Loss 0.626057 Time taken for 1 epoch 261.69165444374084 sec Epoch 12 Batch 0 Loss 0.5977 Epoch 12 Batch 100 Loss 0.5946 Epoch 12 Batch 200 Loss 0.5669 Epoch 12 Batch 300 Loss 0.6404 Epoch 12 Batch 400 Loss 0.6047 Epoch 12 Batch 500 Loss 0.5737 Epoch 12 Batch 600 Loss 0.6484 Epoch 12 Batch 700 Loss 0.6829 Epoch 12 Batch 800 Loss 0.5373 Epoch 12 Batch 900 Loss 0.5781 Epoch 12 Batch 1000 Loss 0.5809 Epoch 12 Batch 1100 Loss 0.6197 Epoch 12 Batch 1200 Loss 0.5552 Epoch 12 Loss 0.607003 Time taken for 1 epoch 262.31527185440063 sec Epoch 13 Batch 0 Loss 0.6487 Epoch 13 Batch 100 Loss 0.6125 Epoch 13 Batch 200 Loss 0.5749 Epoch 13 Batch 300 Loss 0.6284 Epoch 13 Batch 400 Loss 0.5512 Epoch 13 Batch 500 Loss 0.5530 Epoch 13 Batch 600 Loss 0.5457 Epoch 13 Batch 700 Loss 0.5863 Epoch 13 Batch 800 Loss 0.5973 Epoch 13 Batch 900 Loss 0.6671 Epoch 13 Batch 1000 Loss 0.6760 Epoch 13 Batch 1100 Loss 0.5488 Epoch 13 Batch 1200 Loss 0.6886 Epoch 13 Loss 0.587778 Time taken for 1 epoch 263.3172333240509 sec Epoch 14 Batch 0 Loss 0.5918 Epoch 14 Batch 100 Loss 0.6638 Epoch 14 Batch 200 Loss 0.5363 Epoch 14 Batch 300 Loss 0.5382 Epoch 14 Batch 400 Loss 0.6229 Epoch 14 Batch 500 Loss 0.5044 Epoch 14 Batch 600 Loss 0.5102 Epoch 14 Batch 700 Loss 0.6639 Epoch 14 Batch 800 Loss 0.5602 Epoch 14 Batch 900 Loss 0.5231 Epoch 14 Batch 1000 Loss 0.5534 Epoch 14 Batch 1100 Loss 0.5675 Epoch 14 Batch 1200 Loss 0.5104 Epoch 14 Loss 0.569652 Time taken for 1 epoch 262.9858510494232 sec Epoch 15 Batch 0 Loss 0.6006 Epoch 15 Batch 100 Loss 0.5741 Epoch 15 Batch 200 Loss 0.5694 Epoch 15 Batch 300 Loss 0.5299 Epoch 15 Batch 400 Loss 0.6709 Epoch 15 Batch 500 Loss 0.5226 Epoch 15 Batch 600 Loss 0.5265 Epoch 15 Batch 700 Loss 0.7045 Epoch 15 Batch 800 Loss 0.5379 Epoch 15 Batch 900 Loss 0.6596 Epoch 15 Batch 1000 Loss 0.6422 Epoch 15 Batch 1100 Loss 0.5319 Epoch 15 Batch 1200 Loss 0.5121
2024-10-23 22:10:08.272078: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.552805 Time taken for 1 epoch 263.6769394874573 sec Epoch 16 Batch 0 Loss 0.5874 Epoch 16 Batch 100 Loss 0.5352 Epoch 16 Batch 200 Loss 0.6396 Epoch 16 Batch 300 Loss 0.5557 Epoch 16 Batch 400 Loss 0.5130 Epoch 16 Batch 500 Loss 0.6142 Epoch 16 Batch 600 Loss 0.5125 Epoch 16 Batch 700 Loss 0.4839 Epoch 16 Batch 800 Loss 0.4985 Epoch 16 Batch 900 Loss 0.4384 Epoch 16 Batch 1000 Loss 0.5347 Epoch 16 Batch 1100 Loss 0.5782 Epoch 16 Batch 1200 Loss 0.4964 Epoch 16 Loss 0.536064 Time taken for 1 epoch 263.8788628578186 sec Epoch 17 Batch 0 Loss 0.5677 Epoch 17 Batch 100 Loss 0.5209 Epoch 17 Batch 200 Loss 0.5020 Epoch 17 Batch 300 Loss 0.5294 Epoch 17 Batch 400 Loss 0.5991 Epoch 17 Batch 500 Loss 0.5233 Epoch 17 Batch 600 Loss 0.4001 Epoch 17 Batch 700 Loss 0.5331 Epoch 17 Batch 800 Loss 0.4739 Epoch 17 Batch 900 Loss 0.4897 Epoch 17 Batch 1000 Loss 0.5354 Epoch 17 Batch 1100 Loss 0.4603 Epoch 17 Batch 1200 Loss 0.4435 Epoch 17 Loss 0.520028 Time taken for 1 epoch 264.11608695983887 sec Epoch 18 Batch 0 Loss 0.5663 Epoch 18 Batch 100 Loss 0.6248 Epoch 18 Batch 200 Loss 0.5025 Epoch 18 Batch 300 Loss 0.5184 Epoch 18 Batch 400 Loss 0.4410 Epoch 18 Batch 500 Loss 0.4576 Epoch 18 Batch 600 Loss 0.5157 Epoch 18 Batch 700 Loss 0.5508 Epoch 18 Batch 800 Loss 0.4744 Epoch 18 Batch 900 Loss 0.4751 Epoch 18 Batch 1000 Loss 0.5086 Epoch 18 Batch 1100 Loss 0.4914 Epoch 18 Batch 1200 Loss 0.5353 Epoch 18 Loss 0.503900 Time taken for 1 epoch 263.3749485015869 sec Epoch 19 Batch 0 Loss 0.4303 Epoch 19 Batch 100 Loss 0.5605 Epoch 19 Batch 200 Loss 0.4685 Epoch 19 Batch 300 Loss 0.4779 Epoch 19 Batch 400 Loss 0.5489 Epoch 19 Batch 500 Loss 0.4112 Epoch 19 Batch 600 Loss 0.4930 Epoch 19 Batch 700 Loss 0.5393 Epoch 19 Batch 800 Loss 0.4403 Epoch 19 Batch 900 Loss 0.5373 Epoch 19 Batch 1000 Loss 0.4607 Epoch 19 Batch 1100 Loss 0.5792 Epoch 19 Batch 1200 Loss 0.4879 Epoch 19 Loss 0.489598 Time taken for 1 epoch 263.7753279209137 sec Epoch 20 Batch 0 Loss 0.4724 Epoch 20 Batch 100 Loss 0.3602 Epoch 20 Batch 200 Loss 0.4118 Epoch 20 Batch 300 Loss 0.5279 Epoch 20 Batch 400 Loss 0.5263 Epoch 20 Batch 500 Loss 0.5329 Epoch 20 Batch 600 Loss 0.5500 Epoch 20 Batch 700 Loss 0.4676 Epoch 20 Batch 800 Loss 0.4057 Epoch 20 Batch 900 Loss 0.4812 Epoch 20 Batch 1000 Loss 0.5685 Epoch 20 Batch 1100 Loss 0.4559 Epoch 20 Batch 1200 Loss 0.4333 Epoch 20 Loss 0.474137 Time taken for 1 epoch 263.65032482147217 sec
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder_gru_L3_resnet, decoder_gru_L3_resnet)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
display_bleu_score(image, result)
./train2014/COCO_train2014_000000274277.jpg
W0000 00:00:1729715528.379348 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.380787 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.382240 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.383702 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.385150 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.386612 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.388053 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.389501 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.390958 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.392489 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.394050 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.395597 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.397332 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.399031 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.400794 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.402434 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.404337 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.442045 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.443380 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.444755 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.446136 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.447493 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.448877 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.450251 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.451620 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.452995 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.454347 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.455728 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.457082 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.458472 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.459857 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.461231 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.462638 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.463994 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.465399 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.466857 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.468260 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.469628 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.471003 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.472402 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.473859 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.475283 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.476774 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.478212 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.479661 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.481098 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.482513 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.483954 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.485378 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.486803 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.488384 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.489901 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.491446 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.492948 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.494392 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.495812 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.497243 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.550585 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.551931 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.553277 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.554635 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.556008 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.557361 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.558705 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.560081 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.561438 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.562850 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.564226 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.565586 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.566880 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.568253 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.569630 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.571065 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.572453 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.573842 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.575240 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.576615 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.577994 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.579364 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.580733 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.582055 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.583418 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.584712 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.586080 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.587447 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.588861 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.590236 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.591586 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.592962 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.594307 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.595737 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.597161 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.598631 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.599980 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.601457 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.602837 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.604179 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.651458 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.652783 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.654184 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.655568 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.656938 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.658296 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.659668 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.661019 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.662389 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.663760 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.665108 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.666462 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.667828 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.669175 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.670533 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.671881 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.673249 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.674647 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.676079 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.677436 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.678807 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.680249 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.681689 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.683233 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.684623 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.686102 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.687478 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.688918 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.690323 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.691730 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.693118 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.694518 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.695922 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.697358 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.698791 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.700271 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.701708 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.703175 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.704661 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.706182 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.742811 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.744317 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.745741 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.747164 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.748645 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.750107 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.751563 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.752989 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.754453 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.755867 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.757303 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.758789 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.760246 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.761681 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.763114 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.764683 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.766099 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.767561 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.769011 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.770475 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.771955 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.773410 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.774840 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.776294 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.777886 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.779345 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.780828 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.782327 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.783813 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.785225 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.786607 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.788080 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.789555 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.791081 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.792584 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.794057 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.795401 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.796873 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.798341 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.799849 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.857499 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.858889 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.860254 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.861645 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.863030 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.864402 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.865773 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.867117 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.868504 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.869876 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.871235 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.872627 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.873984 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.875383 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.876757 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.878162 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.879567 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.880929 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.882279 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.883670 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.885046 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.886413 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.887794 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.889209 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.890613 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.892000 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.893381 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.894759 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.896183 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.897586 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.898973 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.900275 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.901618 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.903012 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.904444 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.905898 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.907280 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.908740 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.910090 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.911453 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.946828 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.948158 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.949575 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.950961 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.952395 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.953800 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.955163 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.956532 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.957910 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.959385 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.960781 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.962216 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.963628 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.965030 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.966454 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.967863 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.969288 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.970714 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.972131 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.973575 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.975023 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.976517 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.978019 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.979486 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.980916 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.982425 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.983987 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.985504 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.987053 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.988559 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.990069 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.991593 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.993083 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.994614 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.996108 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.997630 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715528.999159 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.000741 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.002477 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.042991 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.044419 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.045957 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.047484 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.049067 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.050652 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.052238 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.053808 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.055357 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.056866 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.058400 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.059927 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.061568 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.063202 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.064738 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.066245 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.067857 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.069420 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.071005 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.072623 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.074162 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.075708 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.077340 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.078871 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.080263 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.081833 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.083361 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.084949 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.086488 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.088036 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.089627 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.091219 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.092561 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.094106 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.095720 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.097315 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.098898 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.100387 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.101801 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.103236 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.183029 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.184468 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.185985 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.187523 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.189081 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.190582 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.192105 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.193627 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.195172 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.196561 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.198091 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.199447 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.201069 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.202661 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.204247 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.205764 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.207321 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.208880 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.210431 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.211958 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.213601 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.215194 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.216789 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.218333 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.219699 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.221242 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.222834 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.224251 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.225812 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.227428 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.228968 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.230568 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.232136 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.233656 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.235047 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.236609 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.238043 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.239678 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.281088 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.282635 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.284260 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.285891 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.287451 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.289015 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.290702 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.292288 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.293962 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.295518 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.297077 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.298948 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.300643 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.302274 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.303974 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.305630 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.307312 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.308904 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.310899 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.312848 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.314548 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.316352 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.318017 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.319724 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.321414 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.323219 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.325029 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.326890 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.328725 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.330527 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.332270 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.334075 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.335756 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.337574 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.339239 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.341106 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.343274 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.385529 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.386993 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.388550 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.390072 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.391526 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.392996 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.394527 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.396075 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.397578 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.399093 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.401365 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.403556 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.405734 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.407837 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.409991 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.412113 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.414205 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.416383 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.417939 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.420041 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.422313 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.423852 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.425959 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.428072 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.430309 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.432576 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.434662 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.436756 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.438953 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.441058 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.443173 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.445343 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.447438 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.449562 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.451671 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.453809 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.455902 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.458106 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729715529.460312 9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
Real Caption: <start> a small child is petting a brown cow by a fence <end> Prediction Caption: a boy is working on a keyboard in kitchen levitating in a blender <end>
************************************************************ Predicted Caption : a boy is working on a keyboard in kitchen levitating in a blender <end> ************************************************************ References : <start> a small child is petting a brown cow by a fence <end> <start> a father holds his daughter and <unk> her pet a cow <end> <start> a young girl is reaching out to pet a large cow <end> <start> a man holding a baby who is touching a cow <end> <start> the young child is petting the cow at the farm <end> ... ************************************************************ BLEU Score : unigram = 0.3571428571 bigram = 0.0000000000 trigram = 0.0000000000 4-gram = 0.0000000000 ************************************************************
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 2-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg) /home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 3-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg) /home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 4-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg)
In [ ]:
encoder_gru_L3_resnet.save('models/captioning_models/encoder_resnet_gru_model.keras')
decoder_gru_L3_resnet.save('models/captioning_models/decoder_resnet_gru_model.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_L3_resnet, decoder_gru_L3_resnet, img_name_val, cap_val, tokenizer)
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.0972967400 bigram = 0.0000000000 trigram = 0.0000000000 4-gram = 0.0000000000 ************************************************************
(0.09729673999997569, 4.261733441016533e-155, 3.7465316748112534e-204, 9.244315369840666e-232)
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
Model 4¶
LSTM with BLEU and ResNet50
In [ ]:
# Instantiate encoder and decoder
encoder_lstm, decoder_lstm = create_model(CNN_Encoder, RNN_Decoder_LSTM, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_lstm,
decoder=decoder_lstm,
optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_lstm, decoder_lstm, ckpt_manager)
/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/base_optimizer.py:664: UserWarning: Gradients do not exist for variables ['kernel', 'kernel', 'kernel', 'bias', 'kernel'] when minimizing the loss. If using `model.compile()`, did you forget to provide a `loss` argument? warnings.warn(
Epoch 1 Batch 0 Loss 1.3831 Epoch 1 Batch 100 Loss 0.9723 Epoch 1 Batch 200 Loss 0.8904 Epoch 1 Batch 300 Loss 0.8751 Epoch 1 Batch 400 Loss 0.8336 Epoch 1 Batch 500 Loss 0.7832 Epoch 1 Batch 600 Loss 0.7403 Epoch 1 Loss 0.870570 Time taken for 1 epoch 152.00215911865234 sec Epoch 2 Batch 0 Loss 0.8049 Epoch 2 Batch 100 Loss 0.8281 Epoch 2 Batch 200 Loss 0.7264 Epoch 2 Batch 300 Loss 0.6799 Epoch 2 Batch 400 Loss 0.6123 Epoch 2 Batch 500 Loss 0.7127 Epoch 2 Batch 600 Loss 0.6841 Epoch 2 Loss 0.738784 Time taken for 1 epoch 93.13927578926086 sec Epoch 3 Batch 0 Loss 0.6805 Epoch 3 Batch 100 Loss 0.6375 Epoch 3 Batch 200 Loss 0.7164 Epoch 3 Batch 300 Loss 0.6625 Epoch 3 Batch 400 Loss 0.5675 Epoch 3 Batch 500 Loss 0.6545 Epoch 3 Batch 600 Loss 0.6307 Epoch 3 Loss 0.678108 Time taken for 1 epoch 93.11038064956665 sec Epoch 4 Batch 0 Loss 0.7152 Epoch 4 Batch 100 Loss 0.6767 Epoch 4 Batch 200 Loss 0.6591 Epoch 4 Batch 300 Loss 0.5683 Epoch 4 Batch 400 Loss 0.6328 Epoch 4 Batch 500 Loss 0.6781 Epoch 4 Batch 600 Loss 0.5984 Epoch 4 Loss 0.633837 Time taken for 1 epoch 93.25185775756836 sec Epoch 5 Batch 0 Loss 0.6055 Epoch 5 Batch 100 Loss 0.5480 Epoch 5 Batch 200 Loss 0.5922 Epoch 5 Batch 300 Loss 0.5487 Epoch 5 Batch 400 Loss 0.5750 Epoch 5 Batch 500 Loss 0.5540 Epoch 5 Batch 600 Loss 0.5647 Epoch 5 Loss 0.597418 Time taken for 1 epoch 92.65495610237122 sec Epoch 6 Batch 0 Loss 0.5252 Epoch 6 Batch 100 Loss 0.5995 Epoch 6 Batch 200 Loss 0.5775 Epoch 6 Batch 300 Loss 0.5506 Epoch 6 Batch 400 Loss 0.5226 Epoch 6 Batch 500 Loss 0.5735 Epoch 6 Batch 600 Loss 0.5934 Epoch 6 Loss 0.565422 Time taken for 1 epoch 99.13107991218567 sec Epoch 7 Batch 0 Loss 0.4687 Epoch 7 Batch 100 Loss 0.5263 Epoch 7 Batch 200 Loss 0.5055 Epoch 7 Batch 300 Loss 0.4717 Epoch 7 Batch 400 Loss 0.5540 Epoch 7 Batch 500 Loss 0.5021 Epoch 7 Batch 600 Loss 0.6039 Epoch 7 Loss 0.536423 Time taken for 1 epoch 93.5994622707367 sec Epoch 8 Batch 0 Loss 0.5791 Epoch 8 Batch 100 Loss 0.5324 Epoch 8 Batch 200 Loss 0.5112 Epoch 8 Batch 300 Loss 0.5310 Epoch 8 Batch 400 Loss 0.5210 Epoch 8 Batch 500 Loss 0.4669 Epoch 8 Batch 600 Loss 0.4941 Epoch 8 Loss 0.509094 Time taken for 1 epoch 93.1858274936676 sec Epoch 9 Batch 0 Loss 0.4531 Epoch 9 Batch 100 Loss 0.4827 Epoch 9 Batch 200 Loss 0.4385 Epoch 9 Batch 300 Loss 0.5299 Epoch 9 Batch 400 Loss 0.4694 Epoch 9 Batch 500 Loss 0.4675 Epoch 9 Batch 600 Loss 0.4686 Epoch 9 Loss 0.484041 Time taken for 1 epoch 93.07637643814087 sec Epoch 10 Batch 0 Loss 0.4952 Epoch 10 Batch 100 Loss 0.4532 Epoch 10 Batch 200 Loss 0.4576 Epoch 10 Batch 300 Loss 0.4407 Epoch 10 Batch 400 Loss 0.4725 Epoch 10 Batch 500 Loss 0.4470 Epoch 10 Batch 600 Loss 0.4543 Epoch 10 Loss 0.460005 Time taken for 1 epoch 93.67944312095642 sec Epoch 11 Batch 0 Loss 0.4750 Epoch 11 Batch 100 Loss 0.4594 Epoch 11 Batch 200 Loss 0.4497 Epoch 11 Batch 300 Loss 0.4565 Epoch 11 Batch 400 Loss 0.4069 Epoch 11 Batch 500 Loss 0.4643 Epoch 11 Batch 600 Loss 0.4308 Epoch 11 Loss 0.436256 Time taken for 1 epoch 99.09716939926147 sec Epoch 12 Batch 0 Loss 0.4806 Epoch 12 Batch 100 Loss 0.4399 Epoch 12 Batch 200 Loss 0.4399 Epoch 12 Batch 300 Loss 0.4013 Epoch 12 Batch 400 Loss 0.3795 Epoch 12 Batch 500 Loss 0.4337 Epoch 12 Batch 600 Loss 0.3773 Epoch 12 Loss 0.415265 Time taken for 1 epoch 94.1980767250061 sec Epoch 13 Batch 0 Loss 0.4185 Epoch 13 Batch 100 Loss 0.4050 Epoch 13 Batch 200 Loss 0.4159 Epoch 13 Batch 300 Loss 0.3988 Epoch 13 Batch 400 Loss 0.3430 Epoch 13 Batch 500 Loss 0.4026 Epoch 13 Batch 600 Loss 0.3719 Epoch 13 Loss 0.395274 Time taken for 1 epoch 108.24577379226685 sec Epoch 14 Batch 0 Loss 0.3550 Epoch 14 Batch 100 Loss 0.3611 Epoch 14 Batch 200 Loss 0.3304 Epoch 14 Batch 300 Loss 0.3564 Epoch 14 Batch 400 Loss 0.3474 Epoch 14 Batch 500 Loss 0.3742 Epoch 14 Batch 600 Loss 0.3846 Epoch 14 Loss 0.375913 Time taken for 1 epoch 94.54929637908936 sec Epoch 15 Batch 0 Loss 0.3404 Epoch 15 Batch 100 Loss 0.3659 Epoch 15 Batch 200 Loss 0.3987 Epoch 15 Batch 300 Loss 0.3594 Epoch 15 Batch 400 Loss 0.3763 Epoch 15 Batch 500 Loss 0.3899 Epoch 15 Batch 600 Loss 0.3747 Epoch 15 Loss 0.357651 Time taken for 1 epoch 94.3346164226532 sec Epoch 16 Batch 0 Loss 0.3317 Epoch 16 Batch 100 Loss 0.3683 Epoch 16 Batch 200 Loss 0.3701 Epoch 16 Batch 300 Loss 0.3195 Epoch 16 Batch 400 Loss 0.3268 Epoch 16 Batch 500 Loss 0.3668 Epoch 16 Batch 600 Loss 0.2972 Epoch 16 Loss 0.341543 Time taken for 1 epoch 99.35520458221436 sec Epoch 17 Batch 0 Loss 0.3200 Epoch 17 Batch 100 Loss 0.3107 Epoch 17 Batch 200 Loss 0.3493 Epoch 17 Batch 300 Loss 0.3187 Epoch 17 Batch 400 Loss 0.2750 Epoch 17 Batch 500 Loss 0.2709 Epoch 17 Batch 600 Loss 0.3122 Epoch 17 Loss 0.326561 Time taken for 1 epoch 93.22835731506348 sec Epoch 18 Batch 0 Loss 0.2596 Epoch 18 Batch 100 Loss 0.3018 Epoch 18 Batch 200 Loss 0.3063 Epoch 18 Batch 300 Loss 0.3251 Epoch 18 Batch 400 Loss 0.3383 Epoch 18 Batch 500 Loss 0.3007 Epoch 18 Batch 600 Loss 0.2908 Epoch 18 Loss 0.311323 Time taken for 1 epoch 93.99069356918335 sec Epoch 19 Batch 0 Loss 0.2561 Epoch 19 Batch 100 Loss 0.2828 Epoch 19 Batch 200 Loss 0.2815 Epoch 19 Batch 300 Loss 0.2784 Epoch 19 Batch 400 Loss 0.2687 Epoch 19 Batch 500 Loss 0.2915 Epoch 19 Batch 600 Loss 0.2500 Epoch 19 Loss 0.298024 Time taken for 1 epoch 93.09543180465698 sec Epoch 20 Batch 0 Loss 0.2711 Epoch 20 Batch 100 Loss 0.2958 Epoch 20 Batch 200 Loss 0.2883 Epoch 20 Batch 300 Loss 0.2930 Epoch 20 Batch 400 Loss 0.2953 Epoch 20 Batch 500 Loss 0.2748 Epoch 20 Batch 600 Loss 0.2866 Epoch 20 Loss 0.286795 Time taken for 1 epoch 92.65783095359802 sec
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder, decoder)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
transfer_learning/train2014/COCO_train2014_000000251219.jpg Real Caption: <start> a kitchen with a refrigerator sink and shelving with various items <end> Prediction Caption: a kitchen with a stove sink tissue and cabinets and stove and appliances <end>
In [ ]:
display_bleu_score(image, result)
************************************************************ Predicted Caption : a kitchen with a stove sink tissue and cabinets and stove and appliances <end> ************************************************************ References : <start> a white refrigerator freezer sitting in a kitchen <end> <start> a white refrigerator that is in a kitchen <end> <start> a photo of household kitchen with a traditional look <end> <start> a modest kitchen with wooden cabinets and a spice shelf <end> <start> a kitchen with a refrigerator sink and shelving with various items <end> ... ************************************************************ BLEU Score : unigram = 0.5714285714 bigram = 0.4193139347 trigram = 0.3119507414 4-gram = 0.2271870978 ************************************************************
In [ ]:
# Save the Encoder model
encoder.save('models/captioning_models/encoder_lstm_resnet.keras')
# Save the Decoder model
decoder.save('models/captioning_models/decoder_lstm_resnet.keras')
In [ ]:
evaluate_average_bleu(encoder, decoder, img_name_val, cap_val, tokenizer)
/usr/local/lib/python3.10/dist-packages/nltk/translate/bleu_score.py:552: UserWarning: The hypothesis contains 0 counts of 2-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg) /usr/local/lib/python3.10/dist-packages/nltk/translate/bleu_score.py:552: UserWarning: The hypothesis contains 0 counts of 3-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg) /usr/local/lib/python3.10/dist-packages/nltk/translate/bleu_score.py:552: UserWarning: The hypothesis contains 0 counts of 4-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg)
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.1137287452 bigram = 0.0000209531 trigram = 0.0000000000 4-gram = 0.0000000000 ************************************************************
(0.11372874523877911, 2.0953124859522655e-05, 1.345579927627796e-106, 9.653423620941904e-159)
Model 5¶
3 Layers GRU with BLEU and InceptionV3
In [ ]:
# Instantiate encoder and decoder
encoder_gru_L3_incv3, decoder_gru_L3_incv3 = create_model(CNN_Encoder, RNN_Decoder_GRU_3L, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_L3_incv3,
decoder=decoder_gru_L3_incv3,
optimizer=optimizer)
checkpoint_path = "./checkpoints/gru-l3-incv3"
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_L3_incv3, decoder_gru_L3_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 2.0273 Epoch 1 Batch 100 Loss 1.4244 Epoch 1 Batch 200 Loss 1.0996 Epoch 1 Batch 300 Loss 1.0915 Epoch 1 Batch 400 Loss 1.0891 Epoch 1 Batch 500 Loss 1.0066 Epoch 1 Batch 600 Loss 0.9345 Epoch 1 Batch 700 Loss 0.9635 Epoch 1 Batch 800 Loss 0.9110 Epoch 1 Batch 900 Loss 0.7981 Epoch 1 Batch 1000 Loss 0.8036 Epoch 1 Batch 1100 Loss 0.8040 Epoch 1 Batch 1200 Loss 0.8165
2024-10-24 10:06:07.386889: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 0.992230 Time taken for 1 epoch 403.0873613357544 sec Epoch 2 Batch 0 Loss 0.8655 Epoch 2 Batch 100 Loss 0.8208 Epoch 2 Batch 200 Loss 0.8260 Epoch 2 Batch 300 Loss 0.8338 Epoch 2 Batch 400 Loss 0.9222 Epoch 2 Batch 500 Loss 0.7434 Epoch 2 Batch 600 Loss 0.8859 Epoch 2 Batch 700 Loss 0.7581 Epoch 2 Batch 800 Loss 0.7558 Epoch 2 Batch 900 Loss 0.9581 Epoch 2 Batch 1000 Loss 0.7948 Epoch 2 Batch 1100 Loss 0.6928 Epoch 2 Batch 1200 Loss 0.6924 Epoch 2 Loss 0.818949 Time taken for 1 epoch 269.3762722015381 sec Epoch 3 Batch 0 Loss 0.8814 Epoch 3 Batch 100 Loss 0.7610 Epoch 3 Batch 200 Loss 0.7679 Epoch 3 Batch 300 Loss 0.7000 Epoch 3 Batch 400 Loss 0.7909 Epoch 3 Batch 500 Loss 0.8011 Epoch 3 Batch 600 Loss 0.8097 Epoch 3 Batch 700 Loss 0.9529 Epoch 3 Batch 800 Loss 0.7974 Epoch 3 Batch 900 Loss 0.8413 Epoch 3 Batch 1000 Loss 0.6295 Epoch 3 Batch 1100 Loss 0.6649 Epoch 3 Batch 1200 Loss 0.7107
2024-10-24 10:15:05.612939: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.763228 Time taken for 1 epoch 268.65580010414124 sec Epoch 4 Batch 0 Loss 0.8183 Epoch 4 Batch 100 Loss 0.7115 Epoch 4 Batch 200 Loss 0.5065 Epoch 4 Batch 300 Loss 0.7208 Epoch 4 Batch 400 Loss 0.7995 Epoch 4 Batch 500 Loss 0.8565 Epoch 4 Batch 600 Loss 0.6849 Epoch 4 Batch 700 Loss 0.7132 Epoch 4 Batch 800 Loss 0.6334 Epoch 4 Batch 900 Loss 0.6796 Epoch 4 Batch 1000 Loss 0.7184 Epoch 4 Batch 1100 Loss 0.6818 Epoch 4 Batch 1200 Loss 0.6700 Epoch 4 Loss 0.727730 Time taken for 1 epoch 269.0176486968994 sec Epoch 5 Batch 0 Loss 0.7593 Epoch 5 Batch 100 Loss 0.7043 Epoch 5 Batch 200 Loss 0.6963 Epoch 5 Batch 300 Loss 0.7011 Epoch 5 Batch 400 Loss 0.7265 Epoch 5 Batch 500 Loss 0.6103 Epoch 5 Batch 600 Loss 0.8109 Epoch 5 Batch 700 Loss 0.6947 Epoch 5 Batch 800 Loss 0.6715 Epoch 5 Batch 900 Loss 0.5703 Epoch 5 Batch 1000 Loss 0.6739 Epoch 5 Batch 1100 Loss 0.5613 Epoch 5 Batch 1200 Loss 0.5914 Epoch 5 Loss 0.700923 Time taken for 1 epoch 270.2045750617981 sec Epoch 6 Batch 0 Loss 0.6572 Epoch 6 Batch 100 Loss 0.5697 Epoch 6 Batch 200 Loss 0.7420 Epoch 6 Batch 300 Loss 0.6157 Epoch 6 Batch 400 Loss 0.9177 Epoch 6 Batch 500 Loss 0.6105 Epoch 6 Batch 600 Loss 0.7113 Epoch 6 Batch 700 Loss 0.6540 Epoch 6 Batch 800 Loss 0.5586 Epoch 6 Batch 900 Loss 0.7040 Epoch 6 Batch 1000 Loss 0.6584 Epoch 6 Batch 1100 Loss 0.6369 Epoch 6 Batch 1200 Loss 0.6958 Epoch 6 Loss 0.673867 Time taken for 1 epoch 270.36888003349304 sec Epoch 7 Batch 0 Loss 0.6811 Epoch 7 Batch 100 Loss 0.6239 Epoch 7 Batch 200 Loss 0.7708 Epoch 7 Batch 300 Loss 0.6421 Epoch 7 Batch 400 Loss 0.6111 Epoch 7 Batch 500 Loss 0.6117 Epoch 7 Batch 600 Loss 0.6681 Epoch 7 Batch 700 Loss 0.6419 Epoch 7 Batch 800 Loss 0.5460 Epoch 7 Batch 900 Loss 0.5885 Epoch 7 Batch 1000 Loss 0.6350 Epoch 7 Batch 1100 Loss 0.6750 Epoch 7 Batch 1200 Loss 0.6399
2024-10-24 10:33:03.323824: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.650628 Time taken for 1 epoch 268.11972188949585 sec Epoch 8 Batch 0 Loss 0.6282 Epoch 8 Batch 100 Loss 0.6848 Epoch 8 Batch 200 Loss 0.6740 Epoch 8 Batch 300 Loss 0.5582 Epoch 8 Batch 400 Loss 0.5874 Epoch 8 Batch 500 Loss 0.6132 Epoch 8 Batch 600 Loss 0.6987 Epoch 8 Batch 700 Loss 0.6518 Epoch 8 Batch 800 Loss 0.5698 Epoch 8 Batch 900 Loss 0.6209 Epoch 8 Batch 1000 Loss 0.6382 Epoch 8 Batch 1100 Loss 0.5934 Epoch 8 Batch 1200 Loss 0.4897 Epoch 8 Loss 0.625871 Time taken for 1 epoch 269.6277639865875 sec Epoch 9 Batch 0 Loss 0.5534 Epoch 9 Batch 100 Loss 0.6635 Epoch 9 Batch 200 Loss 0.7824 Epoch 9 Batch 300 Loss 0.7120 Epoch 9 Batch 400 Loss 0.5911 Epoch 9 Batch 500 Loss 0.5887 Epoch 9 Batch 600 Loss 0.5511 Epoch 9 Batch 700 Loss 0.5679 Epoch 9 Batch 800 Loss 0.4577 Epoch 9 Batch 900 Loss 0.5216 Epoch 9 Batch 1000 Loss 0.5345 Epoch 9 Batch 1100 Loss 0.5544 Epoch 9 Batch 1200 Loss 0.5633 Epoch 9 Loss 0.606553 Time taken for 1 epoch 269.325320482254 sec Epoch 10 Batch 0 Loss 0.6784 Epoch 10 Batch 100 Loss 0.5170 Epoch 10 Batch 200 Loss 0.5869 Epoch 10 Batch 300 Loss 0.4897 Epoch 10 Batch 400 Loss 0.5863 Epoch 10 Batch 500 Loss 0.6526 Epoch 10 Batch 600 Loss 0.6119 Epoch 10 Batch 700 Loss 0.4459 Epoch 10 Batch 800 Loss 0.5375 Epoch 10 Batch 900 Loss 0.5255 Epoch 10 Batch 1000 Loss 0.5257 Epoch 10 Batch 1100 Loss 0.5891 Epoch 10 Batch 1200 Loss 0.5453 Epoch 10 Loss 0.587032 Time taken for 1 epoch 268.5027709007263 sec Epoch 11 Batch 0 Loss 0.5894 Epoch 11 Batch 100 Loss 0.5438 Epoch 11 Batch 200 Loss 0.6820 Epoch 11 Batch 300 Loss 0.6612 Epoch 11 Batch 400 Loss 0.6664 Epoch 11 Batch 500 Loss 0.5198 Epoch 11 Batch 600 Loss 0.5490 Epoch 11 Batch 700 Loss 0.5411 Epoch 11 Batch 800 Loss 0.4953 Epoch 11 Batch 900 Loss 0.6317 Epoch 11 Batch 1000 Loss 0.5701 Epoch 11 Batch 1100 Loss 0.5690 Epoch 11 Batch 1200 Loss 0.5792 Epoch 11 Loss 0.565977 Time taken for 1 epoch 268.39085698127747 sec Epoch 12 Batch 0 Loss 0.4760 Epoch 12 Batch 100 Loss 0.5237 Epoch 12 Batch 200 Loss 0.5141 Epoch 12 Batch 300 Loss 0.5526 Epoch 12 Batch 400 Loss 0.5591 Epoch 12 Batch 500 Loss 0.7082 Epoch 12 Batch 600 Loss 0.6175 Epoch 12 Batch 700 Loss 0.5937 Epoch 12 Batch 800 Loss 0.4652 Epoch 12 Batch 900 Loss 0.5401 Epoch 12 Batch 1000 Loss 0.5784 Epoch 12 Batch 1100 Loss 0.4975 Epoch 12 Batch 1200 Loss 0.5906 Epoch 12 Loss 0.543929 Time taken for 1 epoch 268.069402217865 sec Epoch 13 Batch 0 Loss 0.5277 Epoch 13 Batch 100 Loss 0.5138 Epoch 13 Batch 200 Loss 0.4930 Epoch 13 Batch 300 Loss 0.5789 Epoch 13 Batch 400 Loss 0.4957 Epoch 13 Batch 500 Loss 0.5294 Epoch 13 Batch 600 Loss 0.5152 Epoch 13 Batch 700 Loss 0.5142 Epoch 13 Batch 800 Loss 0.5069 Epoch 13 Batch 900 Loss 0.5006 Epoch 13 Batch 1000 Loss 0.4732 Epoch 13 Batch 1100 Loss 0.4990 Epoch 13 Batch 1200 Loss 0.4456 Epoch 13 Loss 0.523744 Time taken for 1 epoch 268.15740752220154 sec Epoch 14 Batch 0 Loss 0.5118 Epoch 14 Batch 100 Loss 0.5061 Epoch 14 Batch 200 Loss 0.5002 Epoch 14 Batch 300 Loss 0.4504 Epoch 14 Batch 400 Loss 0.4795 Epoch 14 Batch 500 Loss 0.5499 Epoch 14 Batch 600 Loss 0.5171 Epoch 14 Batch 700 Loss 0.4518 Epoch 14 Batch 800 Loss 0.4520 Epoch 14 Batch 900 Loss 0.5086 Epoch 14 Batch 1000 Loss 0.4989 Epoch 14 Batch 1100 Loss 0.4530 Epoch 14 Batch 1200 Loss 0.3852 Epoch 14 Loss 0.504677 Time taken for 1 epoch 267.6439895629883 sec Epoch 15 Batch 0 Loss 0.5873 Epoch 15 Batch 100 Loss 0.5372 Epoch 15 Batch 200 Loss 0.5173 Epoch 15 Batch 300 Loss 0.5251 Epoch 15 Batch 400 Loss 0.5412 Epoch 15 Batch 500 Loss 0.4160 Epoch 15 Batch 600 Loss 0.4662 Epoch 15 Batch 700 Loss 0.4873 Epoch 15 Batch 800 Loss 0.4345 Epoch 15 Batch 900 Loss 0.5547 Epoch 15 Batch 1000 Loss 0.5292 Epoch 15 Batch 1100 Loss 0.5533 Epoch 15 Batch 1200 Loss 0.4562
2024-10-24 11:08:51.620486: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.484940 Time taken for 1 epoch 268.5790066719055 sec Epoch 16 Batch 0 Loss 0.4544 Epoch 16 Batch 100 Loss 0.4405 Epoch 16 Batch 200 Loss 0.5063 Epoch 16 Batch 300 Loss 0.4461 Epoch 16 Batch 400 Loss 0.4816 Epoch 16 Batch 500 Loss 0.4430 Epoch 16 Batch 600 Loss 0.4726 Epoch 16 Batch 700 Loss 0.4879 Epoch 16 Batch 800 Loss 0.4485 Epoch 16 Batch 900 Loss 0.4748 Epoch 16 Batch 1000 Loss 0.4308 Epoch 16 Batch 1100 Loss 0.5551 Epoch 16 Batch 1200 Loss 0.4556 Epoch 16 Loss 0.472030 Time taken for 1 epoch 268.04731154441833 sec Epoch 17 Batch 0 Loss 0.4523 Epoch 17 Batch 100 Loss 0.4665 Epoch 17 Batch 200 Loss 0.5285 Epoch 17 Batch 300 Loss 0.4544 Epoch 17 Batch 400 Loss 0.4492 Epoch 17 Batch 500 Loss 0.5061 Epoch 17 Batch 600 Loss 0.4172 Epoch 17 Batch 700 Loss 0.5060 Epoch 17 Batch 800 Loss 0.4197 Epoch 17 Batch 900 Loss 0.4326 Epoch 17 Batch 1000 Loss 0.4764 Epoch 17 Batch 1100 Loss 0.4239 Epoch 17 Batch 1200 Loss 0.4864 Epoch 17 Loss 0.453266 Time taken for 1 epoch 267.4763150215149 sec Epoch 18 Batch 0 Loss 0.4541 Epoch 18 Batch 100 Loss 0.3888 Epoch 18 Batch 200 Loss 0.4631 Epoch 18 Batch 300 Loss 0.3991 Epoch 18 Batch 400 Loss 0.5477 Epoch 18 Batch 500 Loss 0.5158 Epoch 18 Batch 600 Loss 0.4004 Epoch 18 Batch 700 Loss 0.3718 Epoch 18 Batch 800 Loss 0.4610 Epoch 18 Batch 900 Loss 0.4034 Epoch 18 Batch 1000 Loss 0.4937 Epoch 18 Batch 1100 Loss 0.3711 Epoch 18 Batch 1200 Loss 0.3866 Epoch 18 Loss 0.444814 Time taken for 1 epoch 267.9072439670563 sec Epoch 19 Batch 0 Loss 0.4711 Epoch 19 Batch 100 Loss 0.3697 Epoch 19 Batch 200 Loss 0.4379 Epoch 19 Batch 300 Loss 0.3641 Epoch 19 Batch 400 Loss 0.4565 Epoch 19 Batch 500 Loss 0.4404 Epoch 19 Batch 600 Loss 0.4251 Epoch 19 Batch 700 Loss 0.4838 Epoch 19 Batch 800 Loss 0.4342 Epoch 19 Batch 900 Loss 0.4219 Epoch 19 Batch 1000 Loss 0.4346 Epoch 19 Batch 1100 Loss 0.4481 Epoch 19 Batch 1200 Loss 0.4086 Epoch 19 Loss 0.416508 Time taken for 1 epoch 268.13540625572205 sec Epoch 20 Batch 0 Loss 0.4549 Epoch 20 Batch 100 Loss 0.4114 Epoch 20 Batch 200 Loss 0.4560 Epoch 20 Batch 300 Loss 0.4523 Epoch 20 Batch 400 Loss 0.3907 Epoch 20 Batch 500 Loss 0.3577 Epoch 20 Batch 600 Loss 0.4230 Epoch 20 Batch 700 Loss 0.4810 Epoch 20 Batch 800 Loss 0.4282 Epoch 20 Batch 900 Loss 0.4475 Epoch 20 Batch 1000 Loss 0.3753 Epoch 20 Batch 1100 Loss 0.4372 Epoch 20 Batch 1200 Loss 0.4181 Epoch 20 Loss 0.404555 Time taken for 1 epoch 268.7165153026581 sec
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder_gru_L3_incv3, decoder_gru_L3_incv3)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
display_bleu_score(image, result)
./train2014/COCO_train2014_000000208956.jpg
W0000 00:00:1729762273.326213 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.331408 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.332728 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.334061 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.335404 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.336757 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.338166 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.339525 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.340884 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.342257 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.344827 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.346301 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.347922 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.349417 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.350877 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.352382 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.353856 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.355350 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.356892 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.358480 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.359998 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.361616 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.364751 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.366810 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.368367 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.369962 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.371487 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.373134 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.374740 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.376296 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.414557 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.415975 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.417455 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.418919 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.420397 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.421949 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.423452 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.424957 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.426545 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.428085 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.429882 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.431476 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.433093 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.434716 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.436257 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.437831 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.439446 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.441079 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.442681 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.444349 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.445996 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.447674 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.449217 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.450846 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.452482 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.454020 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.455577 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.457307 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.459043 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.460759 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.462495 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.464181 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.465975 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.467668 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.469421 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.471264 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.473215 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.475287 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.511177 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.512683 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.514173 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.515681 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.517176 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.518674 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.520212 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.521802 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.523411 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.524997 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.526667 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.528298 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.529910 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.531548 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.533212 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.534837 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.536463 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.538220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.539915 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.541751 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.543409 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.545109 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.546862 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.548766 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.550582 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.552400 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.554240 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.555872 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.557515 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.559466 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.561316 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.563061 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.564949 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.566574 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.569902 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.572044 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.573975 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.576223 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.613888 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.615472 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.617098 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.618702 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.620329 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.621943 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.623655 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.625253 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.627021 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.628716 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.630406 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.631999 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.633700 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.635307 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.637038 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.638755 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.640456 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.642156 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.643899 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.645698 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.647813 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.649407 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.651187 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.653032 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.654881 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.656670 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.658477 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.660156 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.661835 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.663764 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.665554 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.667274 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.669210 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.671069 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.672763 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.674726 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.676490 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.678462 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.680314 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.682259 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.726345 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.727624 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.728962 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.730310 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.731669 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.733013 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.734349 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.735702 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.737031 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.738383 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.739749 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.741102 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.742441 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.743788 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.745122 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.746544 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.747930 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.749275 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.750657 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.752035 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.753365 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.754730 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.756078 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.757419 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.758771 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.760130 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.761499 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.762865 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.764232 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.765614 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.766965 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.768327 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.769693 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.771080 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.772490 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.773954 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.775471 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.776905 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.778317 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.779728 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.816947 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.818534 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.819902 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.821291 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.822660 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.824044 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.825474 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.826853 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.828220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.829603 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.830949 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.832327 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.833723 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.835114 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.836567 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.837948 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.839323 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.840680 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.842065 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.843465 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.844859 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.846248 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.847628 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.849001 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.850393 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.851763 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.853150 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.854616 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.856165 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.857561 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.858944 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.860338 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.861728 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.863182 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.864641 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.866035 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.867454 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.868907 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.898119 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.899538 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.900917 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.902276 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.903641 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.905116 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.906500 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.907924 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.909271 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.910649 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.912057 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.913437 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.914888 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.916324 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.917730 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.919245 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.920612 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.922012 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.923397 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.924825 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.926264 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.927671 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.929089 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.930500 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.931912 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.933352 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.934790 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.936194 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.937597 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.939018 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.940466 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.942197 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.943727 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.945391 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.947038 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.948511 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.950029 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.959793 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762273.972320 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.056325 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.057686 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.059119 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.060489 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.061855 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.063210 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.064586 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.065969 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.067302 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.068698 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.070106 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.071441 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.072798 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.074137 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.075520 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.076892 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.078296 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.079803 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.081147 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.082527 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.083943 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.085350 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.086777 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.088176 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.089626 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.091006 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.092434 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.093859 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.095250 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.096669 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.098068 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.099497 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.100874 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.102236 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.103634 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.105108 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.106611 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.108003 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.109406 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.110842 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.157538 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.159067 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.160647 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.162196 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.163784 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.165313 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.166873 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.168522 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.170168 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.171802 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.173468 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.175090 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.176753 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.178317 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.179855 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.181489 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.183316 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.185027 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.186689 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.188311 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.189956 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.191543 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.193168 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.194839 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.196442 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.198055 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.199665 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.201309 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.202907 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.204824 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.206535 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.208276 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.210197 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.212113 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.213915 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.215766 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.217381 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.256908 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.258204 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.259583 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.260943 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.262279 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.263642 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.265017 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.266384 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.267743 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.269094 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.270448 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.271815 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.273204 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.274573 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.275930 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.277325 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.278712 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.280133 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.281526 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.282929 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.284386 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.285789 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.287220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.288613 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.290019 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.291408 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.292782 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.294218 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.295615 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.297012 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.298398 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.299783 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.303277 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.304766 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.306267 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.307704 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.309127 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.310553 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.340111 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.341521 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.342869 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.344209 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.345535 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.346885 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.348241 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.349561 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.350904 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.352243 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.353606 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.354981 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.356335 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.357681 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.359027 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.360407 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.361827 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.363220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.364552 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.365940 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.367296 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.368732 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.370142 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.371505 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.372897 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.374283 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.375753 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.377131 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.378483 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.379827 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.381214 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.382638 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.384124 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.385521 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.386989 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.389133 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.390556 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.391967 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.393381 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.396167 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.435266 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.436655 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.438043 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.439416 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.440767 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.442115 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.443453 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.444851 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.446210 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.447567 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.448917 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.450270 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.451659 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.453064 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.454427 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.455794 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.457158 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.458592 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.459975 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.461402 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.462818 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.464211 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.465607 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.467009 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.468420 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.469858 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.471233 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.472636 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.474018 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.475394 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.476784 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.478198 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.479587 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.481067 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.482540 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.483957 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.485372 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.486781 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.488219 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.491695 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.520779 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.522083 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.523453 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.524792 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.526149 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.527504 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.528868 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.530208 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.531545 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.532901 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.534246 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.535613 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.536978 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.538315 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.539678 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.541082 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.542445 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.543856 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.545222 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.546614 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.548035 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.549441 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.550828 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.552218 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.553620 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.554983 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.556367 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.557767 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.559150 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.560560 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.561973 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.563339 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.564774 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.566254 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.567723 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.569134 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.570544 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.571963 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.607786 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.609107 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.610503 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.611869 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.613270 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.614627 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.616018 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.617380 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.618733 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.620104 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.621487 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.622941 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.624359 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.625726 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.627096 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.628453 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.629820 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.631193 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.632569 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.634030 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.635434 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.636842 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.638249 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.639668 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.641072 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.642461 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.643894 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.645312 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.646700 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.648108 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.649505 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.650905 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.652309 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.653830 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.655326 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.656748 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.658190 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.659642 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.698898 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.700212 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.701587 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.702975 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.704351 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.705716 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.707093 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.708455 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.709820 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.711243 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.712642 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.714016 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.715393 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.716766 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.718161 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.719537 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.720987 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.722363 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.723754 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.725144 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.726562 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.727983 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.729400 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.730836 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.732261 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.733659 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.735063 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.736471 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.737887 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.739309 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.740718 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.742127 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.743653 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.779408 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.780734 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.782091 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.783463 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.784823 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.786170 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.787550 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.788906 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.790256 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.791680 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.793078 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.794445 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.795810 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.797157 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.798523 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.799896 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.801343 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.802719 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.804094 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.805461 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.806875 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.808280 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.809685 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.811108 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.812521 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.813950 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.815391 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.816813 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.818220 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.819658 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.821081 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.822537 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.824067 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.851286 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.852593 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.853957 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.855310 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.856661 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.858005 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.859382 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.860726 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.862072 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.863442 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.864799 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.866218 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.867642 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.868998 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.870361 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.871724 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.873091 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.874469 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.875863 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.877310 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.878712 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.880116 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.881517 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.882924 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.884338 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.885710 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.887161 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.888576 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.890001 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.891408 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.892814 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.894239 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.895660 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.897197 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.898728 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.900159 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.901610 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.903074 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.967380 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.968906 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.970422 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.971879 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.973339 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.974775 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.976228 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.977673 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.979093 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.980559 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.981952 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.983349 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.984794 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.986199 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.987620 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.988987 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.990378 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.991780 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.993246 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.994729 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.996160 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.997593 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762274.999031 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.000500 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.001985 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.003406 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.004937 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.006397 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.007883 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.009332 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.010780 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.012282 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.013779 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.015359 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.016933 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.018391 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.019864 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.021359 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.059599 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.060901 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.062266 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.063666 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.065027 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.066373 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.067761 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.069105 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.070466 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.071868 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.073284 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.074660 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.076028 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.077428 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.078827 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.080203 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.081657 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.083036 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.084451 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.085898 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.087347 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.088765 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.090202 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.091736 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.093217 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.094619 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.096051 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.097473 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.098933 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.100413 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.101843 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.103294 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.104840 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.159623 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.161001 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.162389 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.163754 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.165104 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.166456 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.167833 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.169210 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.170574 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.171949 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.173353 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.174870 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.176268 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.177660 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.179031 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.180417 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.181861 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.183255 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.184671 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.186075 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.187514 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.188980 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.190503 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.191997 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.193473 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.194925 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.196403 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.197864 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.199331 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.200757 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.202218 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.203668 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.205125 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.206582 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.208054 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.209456 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.210904 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.212513 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.214134 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.215585 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.250477 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.251852 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.253254 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.254653 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.256020 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.257402 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.258802 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.260170 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.261551 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.262934 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.264387 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.265839 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.267254 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.268704 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.270085 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.271488 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.272910 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.274337 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.275761 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.277179 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.278678 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.280140 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.281659 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.283116 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.284589 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.286095 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.287613 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.289090 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.290564 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.292034 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.293551 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.294999 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.296485 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.297956 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.299468 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.300896 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.302321 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.303938 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.305555 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.307046 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.346895 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.348342 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.349873 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.351386 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.352872 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.354360 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.355938 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.357460 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.358996 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.360480 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.361978 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.363515 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.365166 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.366752 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.368324 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.369896 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.371475 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.373057 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.374581 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.376270 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.378022 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.379625 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.381335 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.382918 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.384579 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.386181 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.387898 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.389622 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.391400 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.393129 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.394848 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.396594 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.398270 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.399947 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.401512 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.403237 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.404878 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.406628 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.408659 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.456374 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.457691 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.459044 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.460450 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.461834 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.463211 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.464574 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.465925 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.467335 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.468748 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.470141 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.471520 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.472953 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.474398 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.475789 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.477174 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.478554 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.479966 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.481423 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.482850 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.484305 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.485725 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.487176 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.488664 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.490115 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.491543 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.492949 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.494358 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.495819 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.497229 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.498578 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.499926 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.501352 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.502817 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.504349 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.505886 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.507948 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.509417 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.510869 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.541079 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.542388 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.543789 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.545161 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.546531 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.547903 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.549289 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.550680 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.552064 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.553440 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.554844 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.556242 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.557639 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.559040 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.560411 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.561787 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.563175 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.564584 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.565984 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.567391 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.568820 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.570251 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.571716 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.573157 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.574578 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.575994 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.577397 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.578830 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.580256 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.581675 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.583019 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.584363 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.585858 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.587306 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced W0000 00:00:1729762275.588837 9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
Real Caption: <start> a kitchen with a lot of cabinet space and a tile wall <end> Prediction Caption: some modern looking kitchen with an oven and a window <end>
************************************************************ Predicted Caption : some modern looking kitchen with an oven and a window <end> ************************************************************ References : <start> a room showing a kitchen with a microwave and a cooker <end> <start> the lights are on in the kitchen with white cabinets <end> <start> a small kitchen showing counter cabinets and sink <end> <start> a kitchen with a lot of cabinet space and a tile wall <end> <start> a look in to a very typical looking kitchen with white cabinets <end> ... ************************************************************ BLEU Score : unigram = 0.5454545455 bigram = 0.4045199175 trigram = 0.2664896292 4-gram = 0.0000000000 ************************************************************
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 4-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg)
In [ ]:
# Optimizer and Checkpoint Management
checkpoint_path = "./checkpoints/lstm-incv3"
Model 6¶
LSTM and InceptionV3
In [ ]:
# Instantiate encoder and decoder
encoder_lstm_incv3, decoder_lstm_incv3 = create_model(CNN_Encoder, RNN_Decoder_LSTM, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_lstm_incv3,
decoder=decoder_lstm_incv3,
optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_lstm_incv3, decoder_lstm_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 1.9984 Epoch 1 Batch 100 Loss 1.1119 Epoch 1 Batch 200 Loss 1.0133 Epoch 1 Batch 300 Loss 0.9281 Epoch 1 Batch 400 Loss 0.8589 Epoch 1 Batch 500 Loss 0.7671 Epoch 1 Batch 600 Loss 0.8167 Epoch 1 Loss 0.991312 Time taken for 1 epoch 129.95868301391602 sec Epoch 2 Batch 0 Loss 0.9500 Epoch 2 Batch 100 Loss 0.7699 Epoch 2 Batch 200 Loss 0.8934 Epoch 2 Batch 300 Loss 0.7875 Epoch 2 Batch 400 Loss 0.7850 Epoch 2 Batch 500 Loss 0.7493 Epoch 2 Batch 600 Loss 0.7019 Epoch 2 Loss 0.800343 Time taken for 1 epoch 77.09135317802429 sec Epoch 3 Batch 0 Loss 0.7677 Epoch 3 Batch 100 Loss 0.7789 Epoch 3 Batch 200 Loss 0.7544 Epoch 3 Batch 300 Loss 0.7827 Epoch 3 Batch 400 Loss 0.7102 Epoch 3 Batch 500 Loss 0.7426 Epoch 3 Batch 600 Loss 0.7239 Epoch 3 Loss 0.727023 Time taken for 1 epoch 76.90206217765808 sec Epoch 4 Batch 0 Loss 0.8144 Epoch 4 Batch 100 Loss 0.7971 Epoch 4 Batch 200 Loss 0.7578 Epoch 4 Batch 300 Loss 0.6326 Epoch 4 Batch 400 Loss 0.6539 Epoch 4 Batch 500 Loss 0.6167 Epoch 4 Batch 600 Loss 0.6240 Epoch 4 Loss 0.676870 Time taken for 1 epoch 77.05295157432556 sec Epoch 5 Batch 0 Loss 0.5966 Epoch 5 Batch 100 Loss 0.6785 Epoch 5 Batch 200 Loss 0.7144 Epoch 5 Batch 300 Loss 0.6843 Epoch 5 Batch 400 Loss 0.6350 Epoch 5 Batch 500 Loss 0.5858 Epoch 5 Batch 600 Loss 0.5070 Epoch 5 Loss 0.635758 Time taken for 1 epoch 76.85066795349121 sec Epoch 6 Batch 0 Loss 0.6737 Epoch 6 Batch 100 Loss 0.6642 Epoch 6 Batch 200 Loss 0.6621 Epoch 6 Batch 300 Loss 0.5639 Epoch 6 Batch 400 Loss 0.5286 Epoch 6 Batch 500 Loss 0.5816 Epoch 6 Batch 600 Loss 0.5438 Epoch 6 Loss 0.599297 Time taken for 1 epoch 77.16862893104553 sec Epoch 7 Batch 0 Loss 0.6536 Epoch 7 Batch 100 Loss 0.6267 Epoch 7 Batch 200 Loss 0.4939 Epoch 7 Batch 300 Loss 0.5009 Epoch 7 Batch 400 Loss 0.5533 Epoch 7 Batch 500 Loss 0.5848 Epoch 7 Batch 600 Loss 0.5812 Epoch 7 Loss 0.565294 Time taken for 1 epoch 76.95766997337341 sec Epoch 8 Batch 0 Loss 0.5668 Epoch 8 Batch 100 Loss 0.5441 Epoch 8 Batch 200 Loss 0.6014 Epoch 8 Batch 300 Loss 0.4699 Epoch 8 Batch 400 Loss 0.5002 Epoch 8 Batch 500 Loss 0.5269 Epoch 8 Batch 600 Loss 0.5465 Epoch 8 Loss 0.533389 Time taken for 1 epoch 77.0095465183258 sec Epoch 9 Batch 0 Loss 0.5225 Epoch 9 Batch 100 Loss 0.5121 Epoch 9 Batch 200 Loss 0.5204 Epoch 9 Batch 300 Loss 0.5028 Epoch 9 Batch 400 Loss 0.5206 Epoch 9 Batch 500 Loss 0.4632 Epoch 9 Batch 600 Loss 0.5342 Epoch 9 Loss 0.503142 Time taken for 1 epoch 77.04063487052917 sec Epoch 10 Batch 0 Loss 0.4582 Epoch 10 Batch 100 Loss 0.4926 Epoch 10 Batch 200 Loss 0.5259 Epoch 10 Batch 300 Loss 0.4841 Epoch 10 Batch 400 Loss 0.4431 Epoch 10 Batch 500 Loss 0.4098 Epoch 10 Batch 600 Loss 0.4590 Epoch 10 Loss 0.474277 Time taken for 1 epoch 76.95212960243225 sec Epoch 11 Batch 0 Loss 0.4652 Epoch 11 Batch 100 Loss 0.4325 Epoch 11 Batch 200 Loss 0.4948 Epoch 11 Batch 300 Loss 0.4654 Epoch 11 Batch 400 Loss 0.4169 Epoch 11 Batch 500 Loss 0.3937 Epoch 11 Batch 600 Loss 0.4375 Epoch 11 Loss 0.446720 Time taken for 1 epoch 77.14709115028381 sec Epoch 12 Batch 0 Loss 0.4276 Epoch 12 Batch 100 Loss 0.4503 Epoch 12 Batch 200 Loss 0.4158 Epoch 12 Batch 300 Loss 0.4238 Epoch 12 Batch 400 Loss 0.4349 Epoch 12 Batch 500 Loss 0.3745 Epoch 12 Batch 600 Loss 0.4211 Epoch 12 Loss 0.420779 Time taken for 1 epoch 76.95024394989014 sec Epoch 13 Batch 0 Loss 0.4466 Epoch 13 Batch 100 Loss 0.3693 Epoch 13 Batch 200 Loss 0.4657 Epoch 13 Batch 300 Loss 0.4158 Epoch 13 Batch 400 Loss 0.3720 Epoch 13 Batch 500 Loss 0.3582 Epoch 13 Batch 600 Loss 0.3500 Epoch 13 Loss 0.397309 Time taken for 1 epoch 77.05328512191772 sec Epoch 14 Batch 0 Loss 0.3655 Epoch 14 Batch 100 Loss 0.3873 Epoch 14 Batch 200 Loss 0.4350 Epoch 14 Batch 300 Loss 0.3454 Epoch 14 Batch 400 Loss 0.3851 Epoch 14 Batch 500 Loss 0.3204 Epoch 14 Batch 600 Loss 0.3528 Epoch 14 Loss 0.374772 Time taken for 1 epoch 77.13632464408875 sec Epoch 15 Batch 0 Loss 0.3951 Epoch 15 Batch 100 Loss 0.3861 Epoch 15 Batch 200 Loss 0.4298 Epoch 15 Batch 300 Loss 0.3083 Epoch 15 Batch 400 Loss 0.3179 Epoch 15 Batch 500 Loss 0.3784 Epoch 15 Batch 600 Loss 0.3721 Epoch 15 Loss 0.355054 Time taken for 1 epoch 77.06284379959106 sec Epoch 16 Batch 0 Loss 0.3485 Epoch 16 Batch 100 Loss 0.3445 Epoch 16 Batch 200 Loss 0.3333 Epoch 16 Batch 300 Loss 0.3346 Epoch 16 Batch 400 Loss 0.3403 Epoch 16 Batch 500 Loss 0.3445 Epoch 16 Batch 600 Loss 0.3397 Epoch 16 Loss 0.335110 Time taken for 1 epoch 77.70569515228271 sec Epoch 17 Batch 0 Loss 0.3407 Epoch 17 Batch 100 Loss 0.3052 Epoch 17 Batch 200 Loss 0.3157 Epoch 17 Batch 300 Loss 0.3101 Epoch 17 Batch 400 Loss 0.3413 Epoch 17 Batch 500 Loss 0.3351 Epoch 17 Batch 600 Loss 0.2729 Epoch 17 Loss 0.317208 Time taken for 1 epoch 77.05879306793213 sec Epoch 18 Batch 0 Loss 0.3027 Epoch 18 Batch 100 Loss 0.2742 Epoch 18 Batch 200 Loss 0.3092 Epoch 18 Batch 300 Loss 0.2876 Epoch 18 Batch 400 Loss 0.3139 Epoch 18 Batch 500 Loss 0.3329 Epoch 18 Batch 600 Loss 0.2722 Epoch 18 Loss 0.301593 Time taken for 1 epoch 76.95434212684631 sec Epoch 19 Batch 0 Loss 0.3318 Epoch 19 Batch 100 Loss 0.2664 Epoch 19 Batch 200 Loss 0.2541 Epoch 19 Batch 300 Loss 0.2981 Epoch 19 Batch 400 Loss 0.2499 Epoch 19 Batch 500 Loss 0.2749 Epoch 19 Batch 600 Loss 0.2625 Epoch 19 Loss 0.285071 Time taken for 1 epoch 76.92547869682312 sec Epoch 20 Batch 0 Loss 0.3262 Epoch 20 Batch 100 Loss 0.2832 Epoch 20 Batch 200 Loss 0.2784 Epoch 20 Batch 300 Loss 0.2378 Epoch 20 Batch 400 Loss 0.2854 Epoch 20 Batch 500 Loss 0.2705 Epoch 20 Batch 600 Loss 0.2510 Epoch 20 Loss 0.272564 Time taken for 1 epoch 77.00969767570496 sec
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder_lstm_incv3, decoder_lstm_incv3)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
/kaggle/working/train2014/COCO_train2014_000000199598.jpg Real Caption: <start> this is a very large bathroom in an empty house <end> Prediction Caption: a bathroom with a large mirror and a light on <end>
In [ ]:
display_bleu_score(image, result)
************************************************************ Predicted Caption : a bathroom with a large mirror and a light on <end> ************************************************************ References : <start> this an empty master bathroom of a vacant house <end> <start> a bathroom with a large white tub next to a window <end> <start> a bath tub sits in a large bathroom <end> <start> a view of a large bathroom with a huge tub in it <end> <start> this is a very large bathroom in an empty house <end> ... ************************************************************ BLEU Score : unigram = 0.6363636364 bigram = 0.5045249791 trigram = 0.4430498054 4-gram = 0.3816330911 ************************************************************
In [ ]:
# Save the Encoder model
encoder_lstm_incv3.save('/kaggle/working/models/captioning_models/encoder_lstm_incv3.keras')
# Save the Decoder model
decoder_lstm_incv3.save('/kaggle/working/models/captioning_models/decoder_lstm_incv3.keras')
In [ ]:
evaluate_average_bleu(encoder_lstm_incv3, decoder_lstm_incv3, img_name_val, cap_val, tokenizer)
/opt/conda/lib/python3.10/site-packages/nltk/translate/bleu_score.py:490: UserWarning: Corpus/Sentence contains 0 counts of 2-gram overlaps. BLEU scores might be undesirable; use SmoothingFunction(). warnings.warn(_msg)
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.0914357750 bigram = 0.2912023558 trigram = 0.4352421055 4-gram = 0.5266112327 ************************************************************
(0.09143577502154202, 0.29120235581086157, 0.4352421054856086, 0.5266112326833678)
In [ ]:
encoder_gru_L3_incv3.save('models/captioning_models/encoder_IncV3_gru_model.keras')
decoder_gru_L3_incv3.save('models/captioning_models/decoder_IncV3_gru_model.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_L3_incv3, decoder_gru_L3_incv3, img_name_val, cap_val, tokenizer)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 2-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg) /home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: The hypothesis contains 0 counts of 3-gram overlaps. Therefore the BLEU score evaluates to 0, independently of how many N-gram overlaps of lower order it contains. Consider using lower n-gram order or use SmoothingFunction() warnings.warn(_msg)
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.1000155120 bigram = 0.0000000000 trigram = 0.0000000000 4-gram = 0.0000000000 ************************************************************
(0.10001551200368086, 4.456034186751795e-155, 3.901438391495613e-204, 9.59098435557862e-232)
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
Model 7¶
LSTM with BLEU and InceptionV3
In [ ]:
# Optimizer and Checkpoint Management
checkpoint_path = "./checkpoints/lstm-incv3"
In [ ]:
# Instantiate encoder and decoder
encoder_lstm_incv3, decoder_lstm_incv3 = create_model(CNN_Encoder, RNN_Decoder_LSTM, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_lstm_incv3,
decoder=decoder_lstm_incv3,
optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_lstm_incv3, decoder_lstm_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 1.9984 Epoch 1 Batch 100 Loss 1.1119 Epoch 1 Batch 200 Loss 1.0133 Epoch 1 Batch 300 Loss 0.9281 Epoch 1 Batch 400 Loss 0.8589 Epoch 1 Batch 500 Loss 0.7671 Epoch 1 Batch 600 Loss 0.8167 Epoch 1 Loss 0.991312 Time taken for 1 epoch 129.95868301391602 sec Epoch 2 Batch 0 Loss 0.9500 Epoch 2 Batch 100 Loss 0.7699 Epoch 2 Batch 200 Loss 0.8934 Epoch 2 Batch 300 Loss 0.7875 Epoch 2 Batch 400 Loss 0.7850 Epoch 2 Batch 500 Loss 0.7493 Epoch 2 Batch 600 Loss 0.7019 Epoch 2 Loss 0.800343 Time taken for 1 epoch 77.09135317802429 sec Epoch 3 Batch 0 Loss 0.7677 Epoch 3 Batch 100 Loss 0.7789 Epoch 3 Batch 200 Loss 0.7544 Epoch 3 Batch 300 Loss 0.7827 Epoch 3 Batch 400 Loss 0.7102 Epoch 3 Batch 500 Loss 0.7426 Epoch 3 Batch 600 Loss 0.7239 Epoch 3 Loss 0.727023 Time taken for 1 epoch 76.90206217765808 sec Epoch 4 Batch 0 Loss 0.8144 Epoch 4 Batch 100 Loss 0.7971 Epoch 4 Batch 200 Loss 0.7578 Epoch 4 Batch 300 Loss 0.6326 Epoch 4 Batch 400 Loss 0.6539 Epoch 4 Batch 500 Loss 0.6167 Epoch 4 Batch 600 Loss 0.6240 Epoch 4 Loss 0.676870 Time taken for 1 epoch 77.05295157432556 sec Epoch 5 Batch 0 Loss 0.5966 Epoch 5 Batch 100 Loss 0.6785 Epoch 5 Batch 200 Loss 0.7144 Epoch 5 Batch 300 Loss 0.6843 Epoch 5 Batch 400 Loss 0.6350 Epoch 5 Batch 500 Loss 0.5858 Epoch 5 Batch 600 Loss 0.5070 Epoch 5 Loss 0.635758 Time taken for 1 epoch 76.85066795349121 sec Epoch 6 Batch 0 Loss 0.6737 Epoch 6 Batch 100 Loss 0.6642 Epoch 6 Batch 200 Loss 0.6621 Epoch 6 Batch 300 Loss 0.5639 Epoch 6 Batch 400 Loss 0.5286 Epoch 6 Batch 500 Loss 0.5816 Epoch 6 Batch 600 Loss 0.5438 Epoch 6 Loss 0.599297 Time taken for 1 epoch 77.16862893104553 sec Epoch 7 Batch 0 Loss 0.6536 Epoch 7 Batch 100 Loss 0.6267 Epoch 7 Batch 200 Loss 0.4939 Epoch 7 Batch 300 Loss 0.5009 Epoch 7 Batch 400 Loss 0.5533 Epoch 7 Batch 500 Loss 0.5848 Epoch 7 Batch 600 Loss 0.5812 Epoch 7 Loss 0.565294 Time taken for 1 epoch 76.95766997337341 sec Epoch 8 Batch 0 Loss 0.5668 Epoch 8 Batch 100 Loss 0.5441 Epoch 8 Batch 200 Loss 0.6014 Epoch 8 Batch 300 Loss 0.4699 Epoch 8 Batch 400 Loss 0.5002 Epoch 8 Batch 500 Loss 0.5269 Epoch 8 Batch 600 Loss 0.5465 Epoch 8 Loss 0.533389 Time taken for 1 epoch 77.0095465183258 sec Epoch 9 Batch 0 Loss 0.5225 Epoch 9 Batch 100 Loss 0.5121 Epoch 9 Batch 200 Loss 0.5204 Epoch 9 Batch 300 Loss 0.5028 Epoch 9 Batch 400 Loss 0.5206 Epoch 9 Batch 500 Loss 0.4632 Epoch 9 Batch 600 Loss 0.5342 Epoch 9 Loss 0.503142 Time taken for 1 epoch 77.04063487052917 sec Epoch 10 Batch 0 Loss 0.4582 Epoch 10 Batch 100 Loss 0.4926 Epoch 10 Batch 200 Loss 0.5259 Epoch 10 Batch 300 Loss 0.4841 Epoch 10 Batch 400 Loss 0.4431 Epoch 10 Batch 500 Loss 0.4098 Epoch 10 Batch 600 Loss 0.4590 Epoch 10 Loss 0.474277 Time taken for 1 epoch 76.95212960243225 sec Epoch 11 Batch 0 Loss 0.4652 Epoch 11 Batch 100 Loss 0.4325 Epoch 11 Batch 200 Loss 0.4948 Epoch 11 Batch 300 Loss 0.4654 Epoch 11 Batch 400 Loss 0.4169 Epoch 11 Batch 500 Loss 0.3937 Epoch 11 Batch 600 Loss 0.4375 Epoch 11 Loss 0.446720 Time taken for 1 epoch 77.14709115028381 sec Epoch 12 Batch 0 Loss 0.4276 Epoch 12 Batch 100 Loss 0.4503 Epoch 12 Batch 200 Loss 0.4158 Epoch 12 Batch 300 Loss 0.4238 Epoch 12 Batch 400 Loss 0.4349 Epoch 12 Batch 500 Loss 0.3745 Epoch 12 Batch 600 Loss 0.4211 Epoch 12 Loss 0.420779 Time taken for 1 epoch 76.95024394989014 sec Epoch 13 Batch 0 Loss 0.4466 Epoch 13 Batch 100 Loss 0.3693 Epoch 13 Batch 200 Loss 0.4657 Epoch 13 Batch 300 Loss 0.4158 Epoch 13 Batch 400 Loss 0.3720 Epoch 13 Batch 500 Loss 0.3582 Epoch 13 Batch 600 Loss 0.3500 Epoch 13 Loss 0.397309 Time taken for 1 epoch 77.05328512191772 sec Epoch 14 Batch 0 Loss 0.3655 Epoch 14 Batch 100 Loss 0.3873 Epoch 14 Batch 200 Loss 0.4350 Epoch 14 Batch 300 Loss 0.3454 Epoch 14 Batch 400 Loss 0.3851 Epoch 14 Batch 500 Loss 0.3204 Epoch 14 Batch 600 Loss 0.3528 Epoch 14 Loss 0.374772 Time taken for 1 epoch 77.13632464408875 sec Epoch 15 Batch 0 Loss 0.3951 Epoch 15 Batch 100 Loss 0.3861 Epoch 15 Batch 200 Loss 0.4298 Epoch 15 Batch 300 Loss 0.3083 Epoch 15 Batch 400 Loss 0.3179 Epoch 15 Batch 500 Loss 0.3784 Epoch 15 Batch 600 Loss 0.3721 Epoch 15 Loss 0.355054 Time taken for 1 epoch 77.06284379959106 sec Epoch 16 Batch 0 Loss 0.3485 Epoch 16 Batch 100 Loss 0.3445 Epoch 16 Batch 200 Loss 0.3333 Epoch 16 Batch 300 Loss 0.3346 Epoch 16 Batch 400 Loss 0.3403 Epoch 16 Batch 500 Loss 0.3445 Epoch 16 Batch 600 Loss 0.3397 Epoch 16 Loss 0.335110 Time taken for 1 epoch 77.70569515228271 sec Epoch 17 Batch 0 Loss 0.3407 Epoch 17 Batch 100 Loss 0.3052 Epoch 17 Batch 200 Loss 0.3157 Epoch 17 Batch 300 Loss 0.3101 Epoch 17 Batch 400 Loss 0.3413 Epoch 17 Batch 500 Loss 0.3351 Epoch 17 Batch 600 Loss 0.2729 Epoch 17 Loss 0.317208 Time taken for 1 epoch 77.05879306793213 sec Epoch 18 Batch 0 Loss 0.3027 Epoch 18 Batch 100 Loss 0.2742 Epoch 18 Batch 200 Loss 0.3092 Epoch 18 Batch 300 Loss 0.2876 Epoch 18 Batch 400 Loss 0.3139 Epoch 18 Batch 500 Loss 0.3329 Epoch 18 Batch 600 Loss 0.2722 Epoch 18 Loss 0.301593 Time taken for 1 epoch 76.95434212684631 sec Epoch 19 Batch 0 Loss 0.3318 Epoch 19 Batch 100 Loss 0.2664 Epoch 19 Batch 200 Loss 0.2541 Epoch 19 Batch 300 Loss 0.2981 Epoch 19 Batch 400 Loss 0.2499 Epoch 19 Batch 500 Loss 0.2749 Epoch 19 Batch 600 Loss 0.2625 Epoch 19 Loss 0.285071 Time taken for 1 epoch 76.92547869682312 sec Epoch 20 Batch 0 Loss 0.3262 Epoch 20 Batch 100 Loss 0.2832 Epoch 20 Batch 200 Loss 0.2784 Epoch 20 Batch 300 Loss 0.2378 Epoch 20 Batch 400 Loss 0.2854 Epoch 20 Batch 500 Loss 0.2705 Epoch 20 Batch 600 Loss 0.2510 Epoch 20 Loss 0.272564 Time taken for 1 epoch 77.00969767570496 sec
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder_lstm_incv3, decoder_lstm_incv3)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
/kaggle/working/train2014/COCO_train2014_000000199598.jpg Real Caption: <start> this is a very large bathroom in an empty house <end> Prediction Caption: a bathroom with a large mirror and a light on <end>
In [ ]:
display_bleu_score(image, result)
************************************************************ Predicted Caption : a bathroom with a large mirror and a light on <end> ************************************************************ References : <start> this an empty master bathroom of a vacant house <end> <start> a bathroom with a large white tub next to a window <end> <start> a bath tub sits in a large bathroom <end> <start> a view of a large bathroom with a huge tub in it <end> <start> this is a very large bathroom in an empty house <end> ... ************************************************************ BLEU Score : unigram = 0.6363636364 bigram = 0.5045249791 trigram = 0.4430498054 4-gram = 0.3816330911 ************************************************************
In [ ]:
# Save the Encoder model
encoder_lstm_incv3.save('/kaggle/working/models/captioning_models/encoder_lstm_incv3.keras')
# Save the Decoder model
decoder_lstm_incv3.save('/kaggle/working/models/captioning_models/decoder_lstm_incv3.keras')
In [ ]:
evaluate_average_bleu(encoder_lstm_incv3, decoder_lstm_incv3, img_name_val, cap_val, tokenizer)
/opt/conda/lib/python3.10/site-packages/nltk/translate/bleu_score.py:490: UserWarning: Corpus/Sentence contains 0 counts of 2-gram overlaps. BLEU scores might be undesirable; use SmoothingFunction(). warnings.warn(_msg)
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.0914357750 bigram = 0.2912023558 trigram = 0.4352421055 4-gram = 0.5266112327 ************************************************************
(0.09143577502154202, 0.29120235581086157, 0.4352421054856086, 0.5266112326833678)
Model 7¶
GRU with BLEU and InceptionV3
In [ ]:
# Instantiate encoder and decoder
encoder_gru_incv3, decoder_gru_incv3 = create_model(CNN_Encoder, RNN_Decoder_GRU, embedding_dim, units, vocab_size)
# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_incv3,
decoder=decoder_gru_incv3,
optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
checkpoint_path,
max_to_keep=5)
# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_incv3, decoder_gru_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 1.8949 Epoch 1 Batch 100 Loss 0.9383 Epoch 1 Batch 200 Loss 0.7920 Epoch 1 Batch 300 Loss 0.8357 Epoch 1 Batch 400 Loss 0.7962 Epoch 1 Batch 500 Loss 0.7882 Epoch 1 Batch 600 Loss 0.6864 Epoch 1 Loss 0.850178 Time taken for 1 epoch 136.3702175617218 sec Epoch 2 Batch 0 Loss 0.8000 Epoch 2 Batch 100 Loss 0.7817 Epoch 2 Batch 200 Loss 0.7412 Epoch 2 Batch 300 Loss 0.7135 Epoch 2 Batch 400 Loss 0.6566 Epoch 2 Batch 500 Loss 0.7199 Epoch 2 Batch 600 Loss 0.6245 Epoch 2 Loss 0.715335 Time taken for 1 epoch 67.74349689483643 sec Epoch 3 Batch 0 Loss 0.6831 Epoch 3 Batch 100 Loss 0.6933 Epoch 3 Batch 200 Loss 0.6508 Epoch 3 Batch 300 Loss 0.6507 Epoch 3 Batch 400 Loss 0.6452 Epoch 3 Batch 500 Loss 0.6322 Epoch 3 Batch 600 Loss 0.6261 Epoch 3 Loss 0.661079 Time taken for 1 epoch 67.64078164100647 sec Epoch 4 Batch 0 Loss 0.7098 Epoch 4 Batch 100 Loss 0.5877 Epoch 4 Batch 200 Loss 0.5796 Epoch 4 Batch 300 Loss 0.5457 Epoch 4 Batch 400 Loss 0.6382 Epoch 4 Batch 500 Loss 0.5834 Epoch 4 Batch 600 Loss 0.5630 Epoch 4 Loss 0.627894 Time taken for 1 epoch 67.46794772148132 sec Epoch 5 Batch 0 Loss 0.6474 Epoch 5 Batch 100 Loss 0.5899 Epoch 5 Batch 200 Loss 0.6607 Epoch 5 Batch 300 Loss 0.5589 Epoch 5 Batch 400 Loss 0.6181 Epoch 5 Batch 500 Loss 0.5543 Epoch 5 Batch 600 Loss 0.6226 Epoch 5 Loss 0.602535 Time taken for 1 epoch 67.5987856388092 sec Epoch 6 Batch 0 Loss 0.6641 Epoch 6 Batch 100 Loss 0.5438 Epoch 6 Batch 200 Loss 0.5980 Epoch 6 Batch 300 Loss 0.6187 Epoch 6 Batch 400 Loss 0.6684 Epoch 6 Batch 500 Loss 0.5840 Epoch 6 Batch 600 Loss 0.6016 Epoch 6 Loss 0.581196 Time taken for 1 epoch 67.73905658721924 sec Epoch 7 Batch 0 Loss 0.6164 Epoch 7 Batch 100 Loss 0.5608 Epoch 7 Batch 200 Loss 0.5554 Epoch 7 Batch 300 Loss 0.5353 Epoch 7 Batch 400 Loss 0.5888 Epoch 7 Batch 500 Loss 0.5284 Epoch 7 Batch 600 Loss 0.5147 Epoch 7 Loss 0.562486 Time taken for 1 epoch 67.7080078125 sec Epoch 8 Batch 0 Loss 0.4977 Epoch 8 Batch 100 Loss 0.5599 Epoch 8 Batch 200 Loss 0.5521 Epoch 8 Batch 300 Loss 0.5285 Epoch 8 Batch 400 Loss 0.5313 Epoch 8 Batch 500 Loss 0.5224 Epoch 8 Batch 600 Loss 0.5081 Epoch 8 Loss 0.546036 Time taken for 1 epoch 67.9389157295227 sec Epoch 9 Batch 0 Loss 0.5639 Epoch 9 Batch 100 Loss 0.5185 Epoch 9 Batch 200 Loss 0.5130 Epoch 9 Batch 300 Loss 0.4707 Epoch 9 Batch 400 Loss 0.5226 Epoch 9 Batch 500 Loss 0.5308 Epoch 9 Batch 600 Loss 0.5287 Epoch 9 Loss 0.529052 Time taken for 1 epoch 68.5266101360321 sec Epoch 10 Batch 0 Loss 0.5091 Epoch 10 Batch 100 Loss 0.5489 Epoch 10 Batch 200 Loss 0.4935 Epoch 10 Batch 300 Loss 0.4904 Epoch 10 Batch 400 Loss 0.5132 Epoch 10 Batch 500 Loss 0.5473 Epoch 10 Batch 600 Loss 0.5261 Epoch 10 Loss 0.513569 Time taken for 1 epoch 68.12801694869995 sec Epoch 11 Batch 0 Loss 0.5286 Epoch 11 Batch 100 Loss 0.4907 Epoch 11 Batch 200 Loss 0.4971 Epoch 11 Batch 300 Loss 0.4784 Epoch 11 Batch 400 Loss 0.4716 Epoch 11 Batch 500 Loss 0.5123 Epoch 11 Batch 600 Loss 0.5165 Epoch 11 Loss 0.498211 Time taken for 1 epoch 68.70136260986328 sec Epoch 12 Batch 0 Loss 0.4602 Epoch 12 Batch 100 Loss 0.4843 Epoch 12 Batch 200 Loss 0.4787 Epoch 12 Batch 300 Loss 0.4907 Epoch 12 Batch 400 Loss 0.4543 Epoch 12 Batch 500 Loss 0.5139 Epoch 12 Batch 600 Loss 0.4508 Epoch 12 Loss 0.483153 Time taken for 1 epoch 68.926766872406 sec Epoch 13 Batch 0 Loss 0.5246 Epoch 13 Batch 100 Loss 0.4433 Epoch 13 Batch 200 Loss 0.4333 Epoch 13 Batch 300 Loss 0.5092 Epoch 13 Batch 400 Loss 0.5386 Epoch 13 Batch 500 Loss 0.4651 Epoch 13 Batch 600 Loss 0.5008 Epoch 13 Loss 0.467838 Time taken for 1 epoch 68.38380694389343 sec Epoch 14 Batch 0 Loss 0.4670 Epoch 14 Batch 100 Loss 0.4697 Epoch 14 Batch 200 Loss 0.4244 Epoch 14 Batch 300 Loss 0.4041 Epoch 14 Batch 400 Loss 0.4178 Epoch 14 Batch 500 Loss 0.4842 Epoch 14 Batch 600 Loss 0.4065 Epoch 14 Loss 0.453543 Time taken for 1 epoch 67.91882824897766 sec Epoch 15 Batch 0 Loss 0.4450 Epoch 15 Batch 100 Loss 0.4496 Epoch 15 Batch 200 Loss 0.4370 Epoch 15 Batch 300 Loss 0.3641 Epoch 15 Batch 400 Loss 0.4662 Epoch 15 Batch 500 Loss 0.4919 Epoch 15 Batch 600 Loss 0.3833 Epoch 15 Loss 0.440085 Time taken for 1 epoch 67.74384880065918 sec Epoch 16 Batch 0 Loss 0.4519 Epoch 16 Batch 100 Loss 0.3983 Epoch 16 Batch 200 Loss 0.3945 Epoch 16 Batch 300 Loss 0.4124 Epoch 16 Batch 400 Loss 0.4057 Epoch 16 Batch 500 Loss 0.4294 Epoch 16 Batch 600 Loss 0.4456 Epoch 16 Loss 0.425590 Time taken for 1 epoch 68.48339438438416 sec Epoch 17 Batch 0 Loss 0.3461 Epoch 17 Batch 100 Loss 0.3895 Epoch 17 Batch 200 Loss 0.3992 Epoch 17 Batch 300 Loss 0.4126 Epoch 17 Batch 400 Loss 0.3693 Epoch 17 Batch 500 Loss 0.4303 Epoch 17 Batch 600 Loss 0.3836 Epoch 17 Loss 0.414078 Time taken for 1 epoch 68.02987790107727 sec Epoch 18 Batch 0 Loss 0.4320 Epoch 18 Batch 100 Loss 0.3899 Epoch 18 Batch 200 Loss 0.4164 Epoch 18 Batch 300 Loss 0.3417 Epoch 18 Batch 400 Loss 0.4256 Epoch 18 Batch 500 Loss 0.3774 Epoch 18 Batch 600 Loss 0.4188 Epoch 18 Loss 0.399674 Time taken for 1 epoch 67.78015422821045 sec Epoch 19 Batch 0 Loss 0.4493 Epoch 19 Batch 100 Loss 0.3805 Epoch 19 Batch 200 Loss 0.4050 Epoch 19 Batch 300 Loss 0.4056 Epoch 19 Batch 400 Loss 0.4120 Epoch 19 Batch 500 Loss 0.3622 Epoch 19 Batch 600 Loss 0.3953 Epoch 19 Loss 0.387221 Time taken for 1 epoch 67.89082050323486 sec Epoch 20 Batch 0 Loss 0.3754 Epoch 20 Batch 100 Loss 0.3667 Epoch 20 Batch 200 Loss 0.3888 Epoch 20 Batch 300 Loss 0.4120 Epoch 20 Batch 400 Loss 0.3742 Epoch 20 Batch 500 Loss 0.3683 Epoch 20 Batch 600 Loss 0.4101 Epoch 20 Loss 0.374367 Time taken for 1 epoch 68.14571762084961 sec
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])
result, attention_plot = evaluate(image, encoder_gru_incv3, decoder_gru_incv3)
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
/kaggle/working/train2014/COCO_train2014_000000501059.jpg Real Caption: <start> a bunch of cars parked on the side of the road <end> Prediction Caption: a car is parked cars <end>
In [ ]:
display_bleu_score(image, result)
************************************************************ Predicted Caption : a car is parked cars <end> ************************************************************ References : <start> a parking meter on a road with cars parked nearby <end> <start> a bunch of cars parked on the side of the road <end> <start> a view of street from curb with cars parked on side of street rows of parking meters trash bin with trees <end> <start> a street filled with parked cars next to parking meters <end> <start> several cars parked on the side of the road <end> ... ************************************************************ BLEU Score : unigram = 0.2897321390 bigram = 0.1586928282 trigram = 0.2235206825 4-gram = 0.2626168670 ************************************************************
In [ ]:
# Save the Encoder model
encoder_gru_incv3.save('models/captioning_models/encoder_gru_incv3.keras')
# Save the Decoder model
decoder_gru_incv3.save('models/captioning_models/decoder_gru_incv3.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_incv3, decoder_gru_incv3, img_name_val, cap_val, tokenizer)
Processed 0/5003 images Processed 500/5003 images Processed 1000/5003 images Processed 1500/5003 images Processed 2000/5003 images Processed 2500/5003 images Processed 3000/5003 images Processed 3500/5003 images Processed 4000/5003 images Processed 4500/5003 images Processed 5000/5003 images ************************************************************ Average BLEU Score on Validation Dataset: unigram = 0.0912703781 bigram = 0.2830066358 trigram = 0.4199234109 4-gram = 0.5064984794 ************************************************************
(0.0912703780917766, 0.28300663576604557, 0.41992341086909, 0.5064984794036137)